AWS: InvalidSignatureException: Signature not yet current and Kubernetes AWS ALB Ingress controller

By | 03/04/2021
 

One day our developers said that they can not create new AWS LoadBalancers via Ingress on our Kubernetes Dev cluster (AWS EKS).

Balancers are created using AWS ALB Controller, see AWS Elastic Kubernetes Service: running ALB Ingress controller.

Errors in the controller’s logs were:

E0304 07:12:38.595113       1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”no object matching key \”test-namespace/test-ingress\” in local store”  “controller”=”alb-ingress-controller” “request”={“Namespace”:”test-namespace”,”Name”:”test-ingress”}
E0304 07:12:39.610204       1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”failed to build LoadBalancer configuration due to failed to get AWS tags. Error: InvalidSignatureException: Signature not yet current: 20210304T071239Z is still later than 20210304T071056Z (20210304T070556Z + 5 min.)\n\tstatus code: 400, request id: 0ee08317-ca50-4b20-b136-49e6dda2b4ff”  “controller”=”alb-ingress-controller” “request”={“Namespace”:”test-namespace”,”Name”:”test-ingress”}

The text “20210304T071239Z is still later than 20210304T071056Z” gave an idea of what is happening, and googling by the “InvalidSignatureException: Signature not yet current” query lead me to this thread on Github Issues, where the AWS SDK discussing and an issue with time synchronization.

Let’s try to recreate a pod with the ALB Controller – a container in it must start count time over and everything must be working then.

Find a controller’s Deployment:

kk -n kube-system get deploy
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
alb-ingress-controller   1/1     1            1           96d

Scale it to zero:

kk -n kube-system scale deploy alb-ingress-controller --replicas=0
deployment.apps/alb-ingress-controller scaled

And back to one pod:

kk -n kube-system scale deploy alb-ingress-controller --replicas=1
deployment.apps/alb-ingress-controller scaled

Check the pod:

kk -n kube-system get pod | grep alb
alb-ingress-controller-7bd97f99bb-bzzdm   1/1     Running   0          4s

Recreate a testing Ingress, check it:

kk -n test-namespace get ingress
NAME           CLASS    HOSTS   ADDRESS                                                                PORTS   AGE
test-ingress   <none>   *       aadca942-testnamespace-tes-***.us-east-2.elb.amazonaws.com   80      4m53s

“Up and running” (c).

Later I’ve found another post about this error, and the idea above was correct, check the Solving “Signature not yet current” Error When Using AWS in Docker.