AWS: security – Instance Metadata Service v1 vs IMDS v2, Kubernetes Pod and Docker containers

By | 04/24/2023
0 Comment

Instance metadata (IMDS – Instance Metadata Service) – data about an EC2 instance, such as information about AMI, IP, hostname, etc.

You can also add User Data to Instance Metadata to store some parameters, which can then be retrieved inside the instance.

See Instance metadata and user data and Instance metadata categories.

From the beginning of the IMDS, a request/response access model was implemented in AWS, i.e. to get access it was enough to make an HTTP request from the host. Later, a session-oriented system was implemented, when a token had to be obtained for access, and this system received the v2 index. See Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service and Use IMDSv2 .

An example of working with IMDSv1

For example, let’s start EC2, and add some value to User Data:

Now, try curlthe address 169.254.169.254:

[simterm]

$ curl http://169.254.169.254/latest/user-data
somedata: somevalue

[/simterm]

Or get the metadata of the instance:

[simterm]

$ curl http://169.254.169.254/latest/meta-data
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
identity-credentials/
instance-action
instance-id
instance-life-cycle
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id
security-groups
services/

[/simterm]

Or the same, but from a Docker container:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# docker run -ti alpine/curl curl http://169.254.169.254/latest/meta-data/
Unable to find image 'alpine/curl:latest' locally
latest: Pulling from alpine/curl
59bf1c3509f3: Pull complete 
da353f38084f: Pull complete 
05df90dbd213: Pull complete 
Digest: sha256:81372de8c566f2d731bde924bed45230018e6d7c21d051c15e283eb8e06dfa2d
Status: Downloaded newer image for alpine/curl:latest
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
...

[/simterm]

Accessing IMDS from Kubernetes Pod

That is, in the case of Kubernetes, any Pod also can get this data.

To check, let’s create a pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-imds
  namespace: default
spec:
  containers:
  - name: test-imds
    image: alpine/curl:latest
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

Run it:

[simterm]

$ kk apply -f pod-imds.yaml

[/simterm]

And run the same request with curl from this Pod:

[simterm]

$ kk exec -ti test-imds -- curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
autoscaling/
block-device-mapping/
events/
hostname
iam/
...

[/simterm]

IMDS Security Credentials: the issue

Among other data, IMDS can return the Access/Secret keys and token used to access the Instance IAM Role.

Let’s get security credentials:

[simterm]

$ curl http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
{
  "Code" : "Success",
  "LastUpdated" : "2023-03-28T12:48:33Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASI***BND",
  "SecretAccessKey" : "4Yz***8Bx",
  "Token" : "IQo***GaQ=",
  "Expiration" : "2023-03-28T18:51:13Z"
}

[/simterm]

And using these Access/Secret keys, we can do everything that the instance is allowed, and, for example, if the IAM role has the AdminAccess permissions connected to the instance, we can get these rights.

Let’s check – add a role with access to S3 buckets to the instance:

Check the role in the metadata:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# curl http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2023-03-28T13:34:51Z",
  "InstanceProfileArn" : "arn:aws:iam::514***799:instance-profile/IMDSTestS3ReadOnlyAccess",
  "InstanceProfileId" : "AIPAXPNJUS3H7XEB7UT24"
}

[/simterm]

And get the keys and token:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# curl http://169.254.169.254/latest/meta-data/iam/security-credentials/IMDSTestS3ReadOnlyAccess
{
  "Code" : "Success",
  "LastUpdated" : "2023-03-28T13:35:16Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASI***3PJ",
  "SecretAccessKey" : "IfC***t2n",
  "Token" : "IQo***o4a",
  "Expiration" : "2023-03-28T20:09:51Z"
}

[/simterm]

Add them to our working machine in ~/.aws/credentials:

[testiam]
aws_access_key_id = ASI***3PJ
aws_secret_access_key = IfC***t2n
aws_session_token = IQo***o4a

And create a profile in the ~/.aws/config:

[profile testiam]
region = eu-central-1
output = json

Check the access now:

[simterm]

$ aws --profile testiam s3 ls
2022-12-19 16:43:06 cronjob-test
2023-02-14 15:02:28 gitlab-s3-cache-test
...

[/simterm]

Not too good, right?

Access permissions to IMDS

To prevent this, you can simply fully disable IMDS, or switch to the IMDS version 2.

Disabling IMDS

For EC2, it is done through the AWS CLI:

[simterm]

$ aws --profile internal ec2 modify-instance-metadata-options --instance-id i-0b0c0e351255ba78c --http-endpoint disabled
{
    "InstanceId": "i-0b0c0e351255ba78c",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "optional",
        "HttpPutResponseHopLimit": 1,
        "HttpEndpoint": "disabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
}

[/simterm]

Or through the AWS Console – EC2 > Instance Settings > Modify instance metadata options :

And now, when requesting meta-data, we will have the 403 – Forbidden error:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# curl http://169.254.169.254/latest/meta-data/iam/info
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
...

[/simterm]

Switching to IMDS v2

If access is necessary, you can disable the IMDS v1 and use only IMDS v2 by adding the mandatory use of the token with the parameter --http-tokens:

[simterm]

$ aws --profile internal ec2 modify-instance-metadata-options --instance-id i-0b0c0e351255ba78c --http-endpoint enabled --http-tokens required
{
    "InstanceId": "i-0b0c0e351255ba78c",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 1,
        "HttpEndpoint": "enabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
}

[/simterm]

Now we have 401 – Unauthorized error:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# curl http://169.254.169.254/latest/meta-data/iam/info
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
...

[/simterm]

But if we will add a token, everything will work.

Let’s get the token:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`

[/simterm]

Check it:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# echo $TOKEN
AQA***Vyg==

[/simterm]

And now run curl again with the header X-aws-ec2-metadata-token:

[simterm]

root@ip-172-31-30-97:/home/ubuntu# curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2023-03-28T13:34:51Z",
  "InstanceProfileArn" : "arn:aws:iam::514***799:instance-profile/IMDSTestS3ReadOnlyAccess",
  "InstanceProfileId" : "AIPAXPNJUS3H7XEB7UT24"
}

[/simterm]

When using Terraform modules to create Node Groups, pay attention to the options. For example, cloudposse/terraform-aws-eks-node-group has IMDSv2 enabled by default, see Behavior changes.

IMDS v2, and Docker

In the case when containers are used, with IMDSv2 enabled, there may be problems when receiving the token, for example:

[simterm]

root@64cbbd918977:/# curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" 
root@64cbbd918977:/# echo $?
56

[/simterm]

To prevent this, add a parameter http-put-response-hop-limit with a value greater than 1, since a call from a container adds another network hop when passing a request from a client to IMDS: the first is a request from the host itself, and the second is from a container on this host:

[simterm]

$ aws --profile internal ec2 modify-instance-metadata-options --http-endpoint enabled --http-tokens required --http-put-response-hop-limit 2 --instance-id i-0b0c0e351255ba78c
{
    "InstanceId": "i-0b0c0e351255ba78c",
    "InstanceMetadataOptions": {
        "State": "pending",
        "HttpTokens": "required",
        "HttpPutResponseHopLimit": 2,
        "HttpEndpoint": "enabled",
...

[/simterm]

And run again:

[simterm]

root@64cbbd918977:/# TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
root@64cbbd918977:/# curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/info
{
  "Code" : "Success",
  "LastUpdated" : "2023-04-10T10:46:45Z",
  "InstanceProfileArn" : "arn:aws:iam::514***799:instance-profile/IMDSTestS3ReadOnlyAccess",
  "InstanceProfileId" : "AIP***T24"
}

[/simterm]

Done.