We have a new EKS cluster 1.30 on our project, where we want to completely remove the old IRSA with OIDC and start using EKS Pod Identities – see AWS: EKS Pod Identities – a replacement for IRSA? Simplifying IAM access management.
And everything seems to work fine, but when I started deploying our Backend API, its Pods did not start and hung in the ContainerCreating status.
Contents
The problem: Kubernetes Secrets Store CSI Driver and “An IAM role must be associated with service account”
The backend secrets are stored in the AWS Secrets Manager, from where they are synchronized to Kubernetes Secrets with the Kubernetes Secrets Store CSI Driver, and then Backend API Pods create their environment variables using these Kubernetes Secrets. See AWS: Kubernetes – Integrating AWS Secrets Manager and Parameter Store.
When checking the status of a Pod, the is wonderful message“An IAM role must be associated with service account“:
Warning FailedMount 43s (x2 over 2m45s) kubelet MountVolume.SetUp failed for volume "backend-api-secret-class-volume" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod dev-backend-api-ns/backend-api-deployment-65c559d47-bb4dz, err: rpc error: code = Unknown desc = us-east-1: An IAM role must be associated with service account backend-api-sa (namespace: dev-backend-api-ns)
Google leads us to the GitHub issue Pod Identity Association not recognized by secrets store CSI driver, which was opened in December 2023.
It also has a pool-request Add support for Pod Identity Association, which is supposed to fix this issue – but it’s still in the Open status, although a few days ago they added a comment that “We are conducting an initial investigation on this feature request and will share updates soon“.
And the driver is now the latest version – 0.3.9:
$ helm list -n kube-system | grep secret secrets-store-csi-driver kube-system 1 2024-07-18 12:46:20.945937022 +0300 EEST deployed secrets-store-csi-driver-1.4.4 1.4.4 secrets-store-csi-driver-provider-aws kube-system 1 2024-07-18 12:46:23.287242734 +0300 EEST deployed secrets-store-csi-driver-provider-aws-0.3.9
So what to do?
The first option is to add OIDC and the old scheme again. But I really don’t want to do this, because I would like to have a completely new authentication on the new Kubernetes cluster, and not to make crutches that will need to be cut out later.
The second option is to try to switch from the Kubernetes Secrets Store CSI Driver to the External Secrets Operator, which can work with a bunch of different providers – AWS Secrets Manager, Hashicorp Vault, Google Secrets Manager, etc.
In addition, I don’t really like the fact that to create a Kubernetes Secret using the Kubernetes Secrets Store CSI Driver, you need to separately describe objects in its SecretProviderClass, and then actually duplicate them in secretObjects.
External Secrets Operator: an overview
So, the External Secrets Operator (ESO) is able to retrieve secrets from external resources and create regular Kubernetes Secrets.
To access AWS, it uses a standard scheme with ServiceAccount, which means we can create an EKS Pod Identity Association on the AWS IAM Role that will provide access to AWS Secrets Manager.
External Secrets Operator uses two main resources:
SecretStore: describes how to access the secrets – which provider (AWS, Google, Vault, etc.) and authentication, and is created at the level of a separate Kubernetes Namespace for access distribution- there is also a
ClusterSecretStorethat can be created globally and is accessible from any Namespace
- there is also a
ExternalSecret: describes which data to receive from the provider and, if necessary, changes to make
External Secrets Operator has a whole bunch of external providers – see Provider.
In AWS, it can work with both the SecretsManager itself and the ParameterStore.
In addition, the External Secrets Operator can even make changes in the AWS Secrets Manager – but we will only use it to create Kubernetes Secrets, because the secrets in the AWS Secrets Manager are created from each project’s Terraform.
So, our task is:
- install External Secrets Operator
- configure it to access AWS with AWS IAM Role and Kubernetes ServiceAccount using EKS Pod Identities
- and create a Kubernetes Secret that we can connect to the Kubernetes Pod to set the necessary environment variables for our service
Installing External Secrets Operator with Helm
Add a repository:
$ helm repo add external-secrets https://charts.external-secrets.io "external-secrets" has been added to your repositories
The Helm chart itself and the available values are here – external-secrets.
Install the chart with the operator in the ops-external-secrets-ns Namespace:
$ helm install -n ops-external-secrets-ns --create-namespace external-secrets external-secrets/external-secrets
Check the Pods:
$ kk -n ops-external-secrets-ns get pod NAME READY STATUS RESTARTS AGE external-secrets-5859d8dc69-vxhjb 1/1 Running 0 33s external-secrets-cert-controller-5bbb8c4bb8-nmjn9 1/1 Running 0 33s external-secrets-webhook-564cd5b69-r5mmb 1/1 Running 0 33s
And ServiceAccounts:
$ kk -n ops-external-secrets-ns get sa NAME SECRETS AGE default 0 10m external-secrets 0 9m59s external-secrets-cert-controller 0 9m59s external-secrets-webhook 0 9m59s
Here we are interested in the ServiceAccount external-secrets – the operator will use it to access AWS Secrets Manager and Parameter Store, and we will connect it to an EKS cluster with a Pod Identity.
Authentication with AWS IAM
What do we need:
- an IAM Policy with permissions to the Secrets Manager and Parameter Store
- an IAM Role with a Trust Policy for the EKS Pod Identity
- connect IAM Policy to this role
Then the External Secrets Operator Pod will assume this Role through the Kubernetes ServiceAccount and will get access to the secrets.
For now, we’ll stick to the simplest scheme, and then we’ll see how you can further separate access through separate IAM Roles for each SecretStore in different Namespaces.
Creating an IAM Policy
Go to the AWS IAM, create a new IAM Policy, allow only read operations on Secrets Manager and Parameter Store resources:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccessToSecretsManager",
"Effect": "Allow",
"Action": [
"secretsmanager:ListSecrets",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [
"arn:aws:secretsmanager:<AWS_REGION>:<AWS_ACCOUNT_ID>:secret:*"
]
},
{
"Sid": "AllowAccessToParameterStore",
"Effect": "Allow",
"Action": [
"ssm:GetParameters",
"ssm:GetParameter",
"ssm:GetParametersByPath"
],
"Resource": [
"arn:aws:ssm:<AWS_REGION>:<AWS_ACCOUNT_ID>:parameter/*"
]
}
]
}
Save it with the external-secrets-operator-test-policy name:
Creating an IAM Role for EKS Pod Identity
Go to the IAM Roles, create a new Role, select EKS – Pod Identity in the Use case:
Connect the IAM Policy external-secrets-operator-test-policy created above:
Save the new Role as external-secrets-operator-test-role, and in the Trust policy we have "Service": "pods.eks.amazonaws.com":
Creating an EKS Pod Identity Association
Now connect this Role to our EKS cluster atlas-eks-ops-1-30-cluster in the ops-external-secrets-ns Namespace to the Kubernetes ServiceAccount named external-secrets:
$ aws --profile work eks create-pod-identity-association --cluster-name atlas-eks-ops-1-30-cluster \ > --role-arn arn:aws:iam::492***148:role/external-secrets-operator-test-role \ > --namespace ops-external-secrets-ns \ > --service-account external-secrets
Check in the EKS > Access:
Okay – everything is ready here. Now the ESO may access the Secrets and Parameters.
Next, we need a SecretStore, which will describe how to access the AWS Secrets Manager or Parameter Store, and an ExternalSecret, which will actually be responsible for Kubernetes Secrets.
Creating a Kubernetes Secrets from AWS Secrets Manager
Documentation for all values for Operator resources is in the API specification.
Creating a SecretStore
We will test in a separate Namespace ops-test-ns.
Let’s write a manifest for the SecretStore:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: test-secret-store
namespace: ops-test-ns
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
Create the resource:
$ kk apply -f test-secretstore.yml secretstore.external-secrets.io/test-secret-store created
Check it and its STATUS:
$ kk get secretstore NAME AGE STATUS CAPABILITIES READY test-secret-store 29s Valid ReadWrite True
Creating an ExternalSecret
Now we need an ExternalSecret that will use the SecretStore created above.
Let’s create a test secret in AWS Secrets Manager:
Save it as test-aws-secret:
And we have a secret with a string {"secret_key_1":"secret_value_1","secret_key_2":"secret_value_2"}:
First, let’s get the entire string and then see how it can be added to a Kubernetes Secret.
Define a manifest for the ExternalSecret:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: test-external-secret
namespace: ops-test-ns
spec:
refreshInterval: 1h
secretStoreRef:
name: test-secret-store
kind: SecretStore
target:
name: test-kubernetes-secret
creationPolicy: Owner
deletionPolicy: Delete
template:
metadata:
labels:
app: test
data:
- secretKey: key_in_the_kubernetes_secret
remoteRef:
# AWS Secrets Manager secret's name
key: test-aws-secret
Here in the spec:
refreshInterval: How often to check for changes in AWS Secrets ManagersecretStoreRef: Which SecretStore to use for access to AWS Secrets Manager (can be specified separately indata.sourceRef.storeRef)target:name: a Kubernetes Secret name to be createdcreationPolicy: “owner” of Kubernetes Secret:Owner: deletes the Kubernetes Secret if the corresponding ExternalSecret is deletedMerge: does not create a Kubernetes Secret, but changes entries in an existing oneOrphan: leaves the Kubernetes Secret if the corresponding ExternalSecret is deleted
deletionPolicy:Retain: leaves Kubernetes Secret if all fields are deleted in the corresponding AWS Secrets Manager SecretDelete: deletes the Kubernetes Secret if all fields are deleted in the corresponding AWS Secrets Manager SecretMerge: deletes all entries in the Kubernetes Secret if all fields are deleted in the corresponding AWS Secrets Manager Secret, but leaves the Kubernetes Secret itself
template: describes the structure of the Kubernetes Secret to be created – itstype,labels,annotations, etc.
data: describes the relationship between a secret in AWS Secrets Manager and a Kubernetes Secret:secretKey: thekeyname in the Kubernetes SecretremoteRef:key: a name of a secret in AWS Secrets Manager
Create the resource:
$ kk apply -f test-externalsecret.yml externalsecret.external-secrets.io/test-external-secret created
Check its status:
$ kk get externalsecret NAME STORE REFRESH INTERVAL STATUS READY test-external-secret test-secret-store 1h SecretSynced True
STATUS == SecretSynced – that is, the External Secrets Operator was able to get the value from AWS Secrets Manager and create a Kubernetes Secret.
In case of problems, you can look at ESO logs:
$ ktail -n ops-external-secrets-ns -l app.kubernetes.io/instance=external-secrets
Check the Kubernetes Secret itself:
$ kk get secret test-kubernetes-secret -o yaml apiVersion: v1 data: key_in_the_kubernetes_secret: eyJzZWNyZXRfa2V5XzEiOiJzZWNyZXRfdmFsdWVfMSIsInNlY3JldF9rZXlfMiI6InNlY3JldF92YWx1ZV8yIn0= ...
And the “eyJzZWNyZXRfa2V5XzEiOiJzZWNyZXRfdmFsdWVfMSIsInNlY3JldF9rZXlfMiI6InNlY3JldF92YWx1ZV8yIn0=” string gives us the whole secret from AWS Secrets Manager:
$ echo eyJzZWNyZXRfa2V5XzEiOiJzZWNyZXRfdmFsdWVfMSIsInNlY3JldF9rZXlfMiI6InNlY3JldF92YWx1ZV8yIn0= | base64 -d
{"secret_key_1":"secret_value_1","secret_key_2":"secret_value_2"}
But in this form, it’s not very useful to us, so we can do it in another way with the property parameter to the data, in which we specify a specific key from the secret:
data:
- secretKey: secret_key_1_value
remoteRef:
# AWS Secrets Manager secret's name
key: test-aws-secret
property: secret_key_1
Then our Kubernetes Secret will look like this:
$ kk get secret test-kubernetes-secret -o yaml apiVersion: v1 data: secret_key_1_value: c2VjcmV0X3ZhbHVlXzE= ...
Where “c2VjcmV0X3ZhbHVlXzE=” is the value of“secret_value_1“.
Or, instead of describing each key from AWS Secrets Manager, we can use spec.dataFrom instead of spec.data in our ExternalSecret:
...
spec:
refreshInterval: 1h
secretStoreRef:
name: test-secret-store
kind: SecretStore
target:
name: test-kubernetes-secret
creationPolicy: Owner
...
dataFrom:
- extract:
key: test-aws-secret
...
And then our Kubernetes Secret will look like this:
$ kk get secret test-kubernetes-secret -o yaml apiVersion: v1 data: secret_key_1: c2VjcmV0X3ZhbHVlXzE= secret_key_2: c2VjcmV0X3ZhbHVlXzI= ...
Where “c2VjcmV0X3ZhbHVlXzE=” is the value of“secret_value_1”, and “c2VjcmV0X3ZhbHVlXzI=” is the value of “secret_value_2″.
Or we can rename the keys, for example:
...
dataFrom:
- extract:
key: test-aws-secret
rewrite:
- regexp:
source: "secret_key_([0-9])"
target: "SECRET_KEY_${1}"
And the result will be:
$ kk get secret test-kubernetes-secret -o yaml apiVersion: v1 data: SECRET_KEY_1: c2VjcmV0X3ZhbHVlXzE= SECRET_KEY_2: c2VjcmV0X3ZhbHVlXzI= ...
Advanced IAM Permissions per SecretStore
Now let’s see how we can use an IAM Role with a separate IAM Policy at the SecretStore level.
That is, instead of having a single IAM Role with an IAM Policy that provides access to all the secrets in AWS Secrets Manager, and which is used by our External Secrets Operator, we can create a separate IAM Role, connect it to a specific SecretStore in a specific Kubernetes Namespace, and then this SecretStore will have access only to those secrets that are described in the corresponding IAM Policy.
Schematically, this can be represented as follows:
So, what we will do:
- in the external-secrets-operator-test-role, which is connected to the external-secrets ServiceAccount via EKS Pod Identity, disable the IAM Policy that provides access to all AWS Secrets
- create a new IAM Policy external-secrets-operator-test-application-policy, which will grant permission only to a specific secret
- create a new IAM Role external-secrets-operator-test-application-role:
- in the Trust Policy, allow it to perform Assume on behalf of the external-secrets-operator-test-role role and
- connect the IAM Policy external-secrets-operator-test-application-policy to this role
- and add a parameter to the SecretStore with the
role: external-secrets-operator-test-application-role
Then External Secrets will work like this:
- the Kubernetes Pod with External Secrets via EKS Pod Identity executes AssumeRole external-secrets-operator-test-role
- when creating an ExternalSecret, it will use a SecretStore, in which the external-secrets-operator-test-application-role is set
- the ExternalSecret with the external-secrets-operator-test-role will perform the second AssumeRole – “assume” the role of external-secrets-operator-test-application-role with its IAM Policy external-secrets-operator-test-application-policy
- and with this role, it will get access to the secret in AWS Secrets Manager
Let’s go.
In the IAM Role external-secrets-operator-test-role, remove the connected IAM Policy that gave full access to AWS Secrets Manager:
Create a new IAM Policy external-secrets-operator-test-application-policy with access to one specific secret test-aws-secret:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccessToSecretsManager",
"Effect": "Allow",
"Action": [
"secretsmanager:ListSecrets",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [
"arn:aws:secretsmanager:us-east-1:492***148:secret:test-aws-secret*"
]
}
]
}
Save it:
Create a new IAM Role with the Custom trust policy, where we allow to Assume from the external-secrets-operator-test-role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Action": [
"sts:AssumeRole",
"sts:TagSession"
],
"Principal": {
"AWS": "arn:aws:iam::492***148:role/external-secrets-operator-test-role"
}
}
]
}
Connect a new Policy to this Role:
And save the new Role as external-secrets-operator-test-application-role:
Now let’s return to our SecretStore and add the role parameter to the spec.provider.aws:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: test-secret-store
namespace: ops-test-ns
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
role: arn:aws:iam::492***148:role/external-secrets-operator-test-application-role
Update the SecretStore :
$ kk apply -f test-secretstore.yml secretstore.external-secrets.io/test-secret-store configured
To check, let’s delete the old one ExternalSecret:
$ kk delete externalsecret test-external-secret externalsecret.external-secrets.io "test-external-secret" deleted
Create it once again:
$ kk apply -f test-externalsecret.yml externalsecret.external-secrets.io/test-external-secret created
And check its status:
$ kk get externalsecret NAME STORE REFRESH INTERVAL STATUS READY test-external-secret test-secret-store 1h SecretSynced True
Okay, that works.
Now let’s try to use another secret from AWS Secrets Manager –“test/rds/kraken”, to which we did not grant permission in the IAM Policy external-secrets-operator-test-application-policy:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: test-external-secret
namespace: ops-test-ns
spec:
refreshInterval: 1h
secretStoreRef:
name: test-secret-store
kind: SecretStore
target:
name: test-kubernetes-secret
...
dataFrom:
- extract:
key: test/rds/kraken
Deploy, check:
$ kk get externalsecret NAME STORE REFRESH INTERVAL STATUS READY test-external-secret test-secret-store 1h SecretSyncedError False
And now, it has STATUS == SecretSyncedError.
This also can be see in logs – the “external-secrets-operator-test-application-role is not authorized to perform: secretsmanager:GetSecretValue on resource: test/rds/kraken” record:
external-secrets-5859d8dc69-2fgc8:external-secrets {... "msg":"could not get secret data from provider","ExternalSecret":{"name":"test-external-secret","namespace":"ops-test-ns"},"error":"AccessDeniedException: User: arn:aws:sts::492***148:assumed-role/external-secrets-operator-test-application-role/1724248118674412261 is not authorized to perform: secretsmanager:GetSecretValue on resource: test/rds/kraken because no identity-based policy allows the secretsmanager:GetSecretValue action\n\tstatus code: 400 ...}
Conclusions
So far, I’ve really liked External Secrets Operator.
First of all, there is really much less manifest code to create resources.
Second, it works well with EKS Pod Identity.
Third, it is a very flexible system for distributing access to secrets.
The fourth is that you can create Kubernetes Secrets from different providers with a single operator.
And in general, it’s a simpler system, because you don’t need to have a DaemonSet with views on each WorkerNode, as implemented in secrets-store-csi-driver-provider-aws, and you don’t need to mount Volumes to Pods to create Kubernetes Secrets.
It looks really cool, so we will migrate to it.
