We have Grafana Loki for logs and need to connect an AWS IAM Role with AWS IAM Policy, which gives access to an AWS S3 bucket where Loki’s chunks and indexes will be stored.
IAM roles for Kubernetes Pods will work in the same way as IAM roles to EC2 instances: a process inside a Pod makes a request to the AWS API, and the AWS SDK or AWS CLI that makes the request also makes the AssumeRole
request with the IAM Role to be used (see AWS: rotation of user IAM keys, EC2 IAM Roles and Jenkins ).
To check how it will work in Kubernetes, let’s create a test IAM Role, ServiceAccount with the annotation of this role, and a Pod with this ServiceAccount that will use this Role.
Looking ahead – it works in Loki in some alternative reality, that is, even when you connect an already tested ServiceAccount to the Loki Pod – it crashes with errors. But eventually, I made it works even there (about setting up Loki itself with AWS S3 a little later in a separate post).
Documentation – IAM roles for service accounts.
Contents
IAM OIDC identity provider verification
In our case, the EKS cluster is deployed using the Terraform aws_eks_cluster
module and the OpenID Connect (OIDC) provider should already be configured.
Go to the cluster page, and find the OpenID Connect provider URL :
Or from the terminal:
[simterm]
$ aws --profile development --region us-west-2 eks describe-cluster --name dev_data_services --query "cluster.identity.oidc.issuer" --output text https://oidc.eks.us-west-2.amazonaws.com/id/537***A10
[/simterm]
Copy the URL without https://
, and find it in the IAM > Identity providers :
Okay, that’s it.
Creating a Kubernetes ServiceAccount with an AWS IAM role
First, let’s create an IAM policy that gives permissions to the AWS S3 bucket, then an IAM Role with a TrustedPolicy
that allows you to perform the AssumeRole
, using the OIDC identity provider (IDP) of the cluster.
AWS IAM policy
Create a test policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::development-dev-loki-object-store", "arn:aws:s3:::development-dev-loki-object-store/*" ] } ] }
Add it to the AWS IAM:
[simterm]
$ aws --profile development iam create-policy --policy-name test-iam-sa-pod-policy --policy-document file://test-iam-sa-pod-policy.json
[/simterm]
Move on to the role.
AWS IAM role and TrustedPolicy
Find an ARN of your Identity Provider:
The OIDC identity provider URL has already been found earlier:
[simterm]
$ aws --profile development --region us-west-2 eks describe-cluster --name dev_data_services --query "cluster.identity.oidc.issuer" --output text https://oidc.eks.us-west-2.amazonaws.com/id/537***A10
[/simterm]
Creating an IAM Role with the AWS CLI
Create a file with the TustedPolicy
, in its Principal
filed specify the ARN of the IDP, and in the Condition
– the OIDC URL of the cluster, which will be allowed to perform requests to the sts.amazonaws.com :
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::638***021:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/537***A10" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.us-west-2.amazonaws.com/id/537***A10:aud": "sts.amazonaws.com" } } } ] }
Create a role with this TrustedPolicy:
[simterm]
$ aws --profile development iam create-role --role-name test-iam-sa-pod-role --assume-role-policy-document file://test-iam-sa-role-trusted-policy.json
[/simterm]
Connect the IAM Policy which we created above with permissions to the S3 to this IAM Role:
[simterm]
$ aws --profile development iam attach-role-policy --role-name test-iam-sa-pod-role --policy-arn=arn:aws:iam::638***021:policy/test-iam-sa-pod-policy
[/simterm]
Create an IAM Role from the AWS Console
Or we can also do it through the Amazon Console page – choose the type of Web identity, Identity Provider of the cluster, and Audience :
Add the IAM Policy for S3:
Save it:
Copy the ARN of the role:
Creating a Kubernetes ServiceAccount
Create a ServiceAccount manifest, in its annotations
set the ARN of the created role:
--- apiVersion: v1 kind: ServiceAccount metadata: name: test-iam-sa-pod-service-account annotations: eks.amazonaws.com/role-arn: arn:aws:iam::638***021:role/test-iam-sa-pod-role
And let’s go to the Kubernetes Pod.
Running a Kubernetes Pod with a ServiceAccount
Create a manifest of the Pod with the serviceAccountName
of the ServiceAccount, so the complete file will look like this:
apiVersion: v1 kind: ServiceAccount metadata: name: test-iam-sa-pod-service-account annotations: eks.amazonaws.com/role-arn: arn:aws:iam::638***021:role/test-iam-sa-pod-role --- apiVersion: v1 kind: Pod metadata: name: test-iam-sa-pod labels: app: test-iam-sa-app spec: containers: - name: test-iam-sa image: amazon/aws-cli command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] serviceAccountName: test-iam-sa-pod-service-account
In the Pod, we will use a Docker image with AWS CLI with the sleep
command, so it will stay Running after startup.
Deploy the ServiceAccount and Pod:
[simterm]
$ kubectl apply -f test-iam-sa-pod.yaml serviceaccount/test-iam-sa-pod-service-account created pod/test-iam-sa-pod created
[/simterm]
Go into it:
[simterm]
$ kubectl exec -ti test-iam-sa-pod -- bash
[/simterm]
And check access to the basket:
[simterm]
bash-4.2# aws s3 ls development-dev-loki-object-store PRE test/
[/simterm]
Done.