We have a Docker image with the
eksctl tool included.
We also have an ЕС2 with Linux with the
There is an AWS IAM Instance Profile attached to this EC2 with the AdminAccess policy assigned.
On this ЕС2 we have Jenkins running in a Docker container, and it spawns its jobs inside in additional Docker containers (matryoshka, you know 🙂 ).
Amongst others, we have a job to provision Elastic Kubernetes Service which uses the
eksctl tool, see the AWS Elastic Kubernetes Service: — автоматизация создания кластера, часть 2 — Ansible, eksctl for details (in Russian for now, will translate it later).
So, the issue is that if I’m running the
eksctl directly from the EC2 – authentification is passed, all resources are accessible:
Put http://169.254.169.254/latest/api/token: net/http: request canceled (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken
But if run the same from the Docker container – it produces the “Put http://169.254.169.254/latest/api/token: net/http: request canceled (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken” error:
And what this error means at all?
AWS Config and authentification
Another interesting thing I noticed, and that helped me to move in the right direction in my investigation was the AWS CLI credentials and config files: if map it via Docker volumes in the Docker container and it will have working credentials – everything works as expected.
I.e. the issue presents only when
eksctl can not authenticate via ACCESS and SECRET keys.
The config file:
Now, let’s map them to the container:
And here is the error.
AWS EC2 Instance Metadata
The first conspicuous thing is the 169.254.169.254 address.
Why? Because as we can remember that a similar address was used to receive an EC2 instance metadata, for example, it was used in the AWS: IAM users keys rotation, EC2 IAM Roles and Jenkins post.
Let’s go ahead and look at the URI – latest/api/token – here is clearly some kind of a request to obtain an authentification token via EC2 Instance Role – and this is the way it should be: if the
eksctl wasn’t able to get authentification data with an environment variables or config files – it tries to get them with the Instance Profile.
PUT? I remember, it was
GET looks more suitable here (GET me this Auth data!)
After short googling was found that AWS changed its authentification process to obtain an ЕС2 metadata, see the Configuring the instance metadata service and links at the end of this post.
So, let’s try to
GET the data directly from the host :
Okay – it works.
And from the container:
And what we can do now?
Well – we just can use the host’s network instead of using Docker networks!
Let’s try it:
- Retrieving Instance Metadata
- Getting Credentials from EC2 Instance Metadata
- Configuring the Instance Metadata Service
- AWS CLI Configuration Variables
- Serverless using AWS profiles only half working
- Assumed role not found when defined in ~/.aws/config
- AWS Enhances Metadata Service Security with IMDSv2
- Getting started with Version 2 of AWS EC2 Instance Metadata service (IMDSv2)