AWS: eksctl – “Put http://169.254.169.254/latest/api/token: net/http: request canceled”

We have a Docker image with the eksctl tool included.

We also have an ЕС2 with Linux with the eksctl.

There is an AWS IAM Instance Profile attached to this EC2 with the AdminAccess policy assigned.

On this ЕС2 we have Jenkins running in a Docker container, and it spawns its jobs inside in additional Docker containers (matryoshka, you know 🙂 ).

Amongst others, we have a job to provision Elastic Kubernetes Service which uses the eksctl tool, see the AWS Elastic Kubernetes Service: — автоматизация создания кластера, часть 2 — Ansible, eksctl for details (in Russian for now, will translate it later).

So, the issue is that if I’m running the eksctl directly from the EC2 – authentification is passed, all resources are accessible:

[simterm]

$ eksctl --region us-east-2 get cluster
NAME                            REGION
bttrm-eks-dev-0                 us-east-2
bttrm-eks-prod-0                us-east-2
eksctl-bttrm-eks-production-1   us-east-2

[/simterm]

Contents

Put http://169.254.169.254/latest/api/token: net/http: request canceled (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken

But if run the same from the Docker container – it produces the “Put http://169.254.169.254/latest/api/token: net/http: request canceled (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken” error:

[simterm]

admin@jenkins-production:~$ docker run bttrm/kubectl-aws:2.5 eksctl get cluster
[!]  retryable error (RequestError: send request failed
caused by: Put http://169.254.169.254/latest/api/token: net/http: request canceled (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 46.10367ms

[/simterm]

But why?

And what this error means at all?

AWS Config and authentification

Another interesting thing I noticed, and that helped me to move in the right direction in my investigation was the AWS CLI credentials and config files: if map it via Docker volumes in the Docker container and it will have working credentials – everything works as expected.

I.e. the issue presents only when eksctl can not authenticate via ACCESS and SECRET keys.

The config file:

[simterm]

admin@jenkins-production:~$ cat .aws/config 
[default]
output = json
region = us-east-2

[/simterm]

Secrets:

[simterm]

admin@jenkins-production:~$ cat .aws/credentials 
[default]
aws_access_key_id = AKI***D4Q
aws_secret_access_key = QUC***BTI

[/simterm]

Now, let’s map them to the container:

[simterm]

admin@jenkins-production:~$ docker run -v /home/admin/.aws:/root/.aws/ bttrm/kubectl-aws:2.5 eksctl get cluster
NAME                            REGION
bttrm-eks-dev-0                 us-east-2
bttrm-eks-prod-0                us-east-2
eksctl-bttrm-eks-production-1   us-east-2

[/simterm]

All works.

Wittout them:

[simterm]

admin@jenkins-production:~$ docker run bttrm/kubectl-aws:2.5 eksctl get cluster
[!]  retryable error (RequestError: send request failed
caused by: Put http://169.254.169.254/latest/api/token: net/http: request canceled (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 32.822673ms

[/simterm]

And here is the error.

AWS EC2 Instance Metadata

The first conspicuous thing is the 169.254.169.254 address.

Why? Because as we can remember that a similar address was used to receive an EC2 instance metadata, for example, it was used in the AWS: IAM users keys rotation, EC2 IAM Roles and Jenkins post.

Let’s go ahead and look at the URI – latest/api/token – here is clearly some kind of a request to obtain an authentification token via EC2 Instance Role – and this is the way it should be: if the eksctl wasn’t able to get authentification data with an environment variables or config files – it tries to get them with the Instance Profile.

But why PUT? I remember, it was GET? And GET looks more suitable here (GET me this Auth data!)

After short googling was found that AWS changed its authentification process to obtain an ЕС2 metadata, see the Configuring the instance metadata service and links at the end of this post.

So, let’s try to GET the data directly from the host :

[simterm]

admin@jenkins-production:~$ curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
AQA***Cyg==

[/simterm]

Okay – it works.

And from the container:

[simterm]

admin@jenkins-production:~$ docker run bttrm/kubectl-aws:2.5 curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
...
curl: (56) Recv failure: Connection reset by peer

[/simterm]

And nothing…

But why?

And what we can do now?

Well – we just can use the host’s network instead of using Docker networks!

Let’s try it:

[simterm]

admin@jenkins-production:~$ docker run --net=host bttrm/kubectl-aws:2.5 curl -sX PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
AQA***ZmA==

[/simterm]

Voila!

It’s ready.