One day our developers said that they can not create new AWS LoadBalancers via Ingress on our Kubernetes Dev cluster (AWS EKS).
Balancers are created using AWS ALB Controller, see AWS Elastic Kubernetes Service: running ALB Ingress controller.
Errors in the controller’s logs were:
E0304 07:12:38.595113 1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”no object matching key \”test-namespace/test-ingress\” in local store” “controller”=”alb-ingress-controller” “request”={“Namespace”:”test-namespace”,”Name”:”test-ingress”}
E0304 07:12:39.610204 1 controller.go:217] kubebuilder/controller “msg”=”Reconciler error” “error”=”failed to build LoadBalancer configuration due to failed to get AWS tags. Error: InvalidSignatureException: Signature not yet current: 20210304T071239Z is still later than 20210304T071056Z (20210304T070556Z + 5 min.)\n\tstatus code: 400, request id: 0ee08317-ca50-4b20-b136-49e6dda2b4ff” “controller”=”alb-ingress-controller” “request”={“Namespace”:”test-namespace”,”Name”:”test-ingress”}
The text “20210304T071239Z is still later than 20210304T071056Z” gave an idea of what is happening, and googling by the “InvalidSignatureException: Signature not yet current” query lead me to this thread on Github Issues, where the AWS SDK discussing and an issue with time synchronization.
Let’s try to recreate a pod with the ALB Controller – a container in it must start count time over and everything must be working then.
Find a controller’s Deployment:
[simterm]
$ kk -n kube-system get deploy NAME READY UP-TO-DATE AVAILABLE AGE alb-ingress-controller 1/1 1 1 96d
[/simterm]
Scale it to zero:
[simterm]
$ kk -n kube-system scale deploy alb-ingress-controller --replicas=0 deployment.apps/alb-ingress-controller scaled
[/simterm]
And back to one pod:
[simterm]
$ kk -n kube-system scale deploy alb-ingress-controller --replicas=1 deployment.apps/alb-ingress-controller scaled
[/simterm]
Check the pod:
[simterm]
$ kk -n kube-system get pod | grep alb alb-ingress-controller-7bd97f99bb-bzzdm 1/1 Running 0 4s
[/simterm]
Recreate a testing Ingress, check it:
[simterm]
$ kk -n test-namespace get ingress NAME CLASS HOSTS ADDRESS PORTS AGE test-ingress <none> * aadca942-testnamespace-tes-***.us-east-2.elb.amazonaws.com 80 4m53s
[/simterm]
“Up and running” (c).
Later I’ve found another post about this error, and the idea above was correct, check the Solving “Signature not yet current” Error When Using AWS in Docker.
