Storing access data in Kubernetes Secrets has an important drawback, because they are only available within the Kubernetes cluster itself.
To make them available to external services, we can use Hashicorp Vault and integrate it with Kubernetes using solutions such as vault-k8s
or use services from AWS – Secrets Manager or Parameter Store.
Integrating AWS Secrets Manager and Parameter Store into Kubernetes will give us the ability to create a new type of resource – SecretProviderClass
which we can connect to Kubernetes Pods as files or environment variables.
For this, we will need AWS Secrets and Configuration Provider (ASCP) and Kubernetes Secrets Store CSI Driver.
Contents
AWS Secrets and Configuration Provider vs Hashicorp Vault
I haven’t used Vault for a long time, but regarding the question “What to use”, it’s a choice between the “setup, configure, and manage Hashicorp Vault yourself” (installation of the Helm chart and configuration of accesses) or “use a ready-made solution from AWS” (in fact, you only need to configure IAM roles).
Also, keep in mind that using AWS services (surprise!) is paid, so if you plan to have thousands of secrets, it’s probably better to use Vault.
In addition, Vault itself provides much more opportunities, for example, the generation of temporary tokens for services, plus, as far as I remember – Kubernetes Pods can receive parameters from Vault without the need to create Kubernetes Secrets, while when using AWS Secrets and Configuration Provider and Kubernetes Secrets Store, CSI Driver for connecting variables will create Kubernetes Secrets.
However, our project already uses Secrets Manager and Parameter Store, so I don’t see a point in Vault (yet), so let’s integrate our existing secrets into an AWS Elastic Kubernetes Service cluster.
AWS Secrets Manager vs Parameter Store
You can read more about the difference between them here – AWS – Difference between Secrets Manager and Parameter Store (Systems Manager), here just briefly.
Common features:
- both use AWS KMS to encrypt data
- both are Key/Value Store
- both support versioning
Key differences:
- Cost:
- Secrets Manager: Charges $0.40 for each secret and $0.05 for every 10,000 API requests
- Parameter Store: for the Standard, it does not take money for storage, with higher throughput it costs $0.05 for every 10,000 API requests, with Advanced parameters – $0.05 for storage and $0.05 for every 10,000 API requests
- Secrets rotation:
- Secrets Manager: has a built-in rotation mechanism and integrates it with services (RDS, DocumentDB, etc)
- Parameter Store: You must implement the rotation yourself
- Cross-account Access:
- Secrets Manager: Supports
- Parameter Store: Not supported
- Cross-Regions Replication:
- Secrets Manager: Supports
- Parameter Store: Not supported
- Data limits:
- Secrets Manager: up to 10KB per secret
- Parameter Store: 4KB for each record (8KB for Advanced Parameters)
- Quantity limits:
- Secrets Manager: 500,000 per region and account
- Parameter Store: 10,000 per region and account
Installing Secrets Store CSI Driver
So, for integration, we need two services – Secrets Store CSI Driver and AWS Secrets and Configuration Provider.
First, we’ll add the Secrets Store CSI Driver.
With its help, we will be able to connect secrets/parameters from AWS as files or variables to Kubernetes Pods.
Add a Helm chart and install it with the syncSecret.enabled=true
parameter to create Kubernetes Secrets and synchronize them with AWS secrets during data rotation (see Sync as Kubernetes Secret ):
[simterm]
$ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts $ helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true
[/simterm]
Check Pods:
[simterm]
$ kubectl -n kube-system get pod | grep secret csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 31s csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 31s
[/simterm]
Installing AWS Secrets and Configuration Provider
Add a repository and install the Helm chart:
[simterm]
$ helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws $ helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws
[/simterm]
Check Pods:
[simterm]
$ kubectl -n kube-system get pod | grep secret csi-secrets-store-secrets-store-csi-driver-kzmcx 3/3 Running 0 9m csi-secrets-store-secrets-store-csi-driver-t7bqc 3/3 Running 0 9m secrets-provider-aws-secrets-store-csi-driver-provider-awskq5g8 1/1 Running 0 23s secrets-provider-aws-secrets-store-csi-driver-provider-awsksq9d 1/1 Running 0 23s
[/simterm]
And look at the CSIDriver:
[simterm]
$ kubectl get csidriver NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE ebs.csi.aws.com true false false <unset> false Persistent 46h efs.csi.aws.com false false false <unset> false Persistent 4d secrets-store.csi.k8s.io false true false <unset> false Ephemeral 10m
[/simterm]
Next, we will configure IAM for IRSA.
IAM Policy та IAM Role для ServiceAccount
In order for Kubernetes Pods to be able to access AWS SecretManager and Parameter Store, we will use IRSA – create a ServiceAacount that will use an IAM Role with an IAM Policy that will have permissions to call Secrets Manager and Parameter Store (see AWS: EKS, OpenID Connect, and ServiceAccounts).
Describe an IAM policy:
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue", "ssm:DescribeParameters", "ssm:GetParameter", "ssm:GetParameters", "ssm:GetParametersByPath" ], "Effect": "Allow", "Resource": "*" } ] }
Create it in the IAM:
[simterm]
$ aws iam create-policy --policy-name ascp-iam-policy --policy-document file://ascp-policy.json { "Policy": { "PolicyName": "ascp-policy", "PolicyId": "ANPAXFIUAIGSBPFEDKZZT", "Arn": "arn:aws:iam::492***148:policy/ascp-iam-policy", ...
[/simterm]
Describe the Trust policy for the IAM Role:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRoleWithWebIdentity", "Principal": { "Federated": "arn:aws:iam::492***148:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/2DC***124" }, "Condition": { "StringEquals": { "oidc.eks.us-east-1.amazonaws.com/id/2DC***124:aud": "sts.amazonaws.com", "oidc.eks.us-east-1.amazonaws.com/id/2DC***124:sub": "system:serviceaccount:default:ascp-test-serviceaccount" } } } ] }
Create the Role itself with this trust policy:
[simterm]
aws iam create-role --role-name ascp-iam-role --assume-role-policy-document file://ascp-trust.json { "Role": { "Path": "/", "RoleName": "ascp-iam-role", "RoleId": "AROAXFIUAIGSLDCB3L4AR", "Arn": "arn:aws:iam::492***148:role/ascp-iam-role", ...
[/simterm]
Attach the ascp-iam-policy
policy to that role:
[simterm]
$ aws iam attach-role-policy --role-name ascp-iam-role --policy-arn=arn:aws:iam::492***148:policy/ascp-iam-policy
[/simterm]
Now we can create a SecretProviderClass and a Pod that will use it.
Create a SecretProviderClass
Let’s add a SecretProviderClass, which will receive a string from the Secrets Manager and a string from the Parameter Store, and then we will connect them to a Kubernetes Pod.
Create a secret in Secrets Manager:
[simterm]
$ aws secretsmanager create-secret --name ascp-secret-test-string --secret-string "secretLine" { "ARN": "arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-string-DNweNg", "Name": "ascp-secret-test-string", "VersionId": "9d4f490d-edcc-4ee0-b43d-5b4e25fa271b" }
[/simterm]
Create a record in the Parameter Store:
[simterm]
$ aws ssm put-parameter --name ascp-ssm-test-param --value 'paramLine' --type "String" { "Version": 1, "Tier": "Standard" }
[/simterm]
Next, describe a SecretProviderClass with two objects
– in the parameters.objects.objectName
set a name of the object in the Secrets Manager or Parameter Store and in the objectType
set where we get this object from:
--- apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: aspc-test-secret-class spec: provider: aws parameters: objects: | - objectName: "ascp-test-string" objectType: "secretsmanager" - objectName: "ascp-ssm-test-param" objectType: "ssmparameter"
Let’s go to the Pod.
Attaching a SecretProviderClass to a Pod as a file
Add a ServiceAccount with the IAM role we created earlier, and a Pod with this ServiceAccount:
--- apiVersion: v1 kind: ServiceAccount metadata: name: ascp-test-serviceaccount namespace: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::492***148:role/ascp-iam-role --- apiVersion: v1 kind: Pod metadata: name: ascp-test-pod spec: containers: - name: my-aws-cli image: amazon/aws-cli:latest command: ['sleep', '36000'] volumeMounts: - name: ascp-test-secret-volume mountPath: /mnt/ascp-secret readOnly: true restartPolicy: Never serviceAccountName: ascp-test-serviceaccount volumes: - name: ascp-test-secret-volume csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: aspc-test-secret-class
Deploy it:
[simterm]
$ kubectl apply -f ascp-test.yaml serviceaccount/ascp-test-serviceaccount created secretproviderclass.secrets-store.csi.x-k8s.io/aspc-test-secret-class created pod/ascp-test-pod created
[/simterm]
Check the Pod:
[simterm]
$ kk describe pod ascp-test-pod ... Mounts: /mnt/ascp-secret from ascp-test-secret-volume (ro) ... Volumes: ... ascp-test-secret-volume: Type: CSI (a Container Storage Interface (CSI) volume source) Driver: secrets-store.csi.k8s.io FSType: ReadOnly: true VolumeAttributes: secretProviderClass=aspc-test-secret-class ...
[/simterm]
And a content of the /mnt/ascp-secret
directory:
[simterm]
$ kk exec -ti ascp-test-pod -- ls -l /mnt/ascp-secret total 8 -rw-r--r-- 1 root root 10 Jul 17 09:32 ascp-secret-test-string -rw-r--r-- 1 root root 9 Jul 17 09:32 ascp-ssm-test-param
[/simterm]
And files there:
[simterm]
$ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-secret-test-string secretLine $ kk exec -ti ascp-test-pod -- cat /mnt/ascp-secret/ascp-ssm-test-param paramLine
[/simterm]
Attaching a SecretProviderClass to a Pod as an environment variable
Mounting secrets as a file may be a good solution for some .env
files, but what about environment variables? For example to pass a DB_PASSWORD
.
To do so, we can add a secretObjects
to the SecretProviderClass – then Kubernetes Secrets Store CSI Driver will create a Kubernetes Secret, which we can connect to the Pod:
--- apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: aspc-test-secret-class spec: provider: aws parameters: objects: | - objectName: "ascp-secret-test-string" objectType: "secretsmanager" - objectName: "ascp-ssm-test-param" objectType: "ssmparameter" secretObjects: - secretName: aspc-test-kube-secret type: Opaque data: - objectName: ascp-secret-test-string key: kube-secret-key
Here:
secretObjects.secretName
: a name of the Kubernetes Secret to be createdsecretObjects.secretName.data.objectName
: must be the same asparameters.objects.objectName
secretObjects.secretName.data.key
: a key for the Kubernetes Secret –data.kube-secret-key
Deploy, and check Kubernetes Secret:
[simterm]
$ kk get secret aspc-test-kube-secret -o yaml apiVersion: v1 data: kube-secret-key: c2VjcmV0TGluZQ== ...
[/simterm]
And a value of the kube-secret-key
:
[simterm]
$ echo c2VjcmV0TGluZQ== | base64 -d secretLine
[/simterm]
Now, let’s connect it to the Pod – add spec.containers.env
with the valueFrom.secretKeyRef
:
--- apiVersion: v1 kind: Pod metadata: name: ascp-test-pod spec: containers: - name: my-aws-cli image: amazon/aws-cli:latest command: ['sleep', '36000'] env: - name: SECRET valueFrom: secretKeyRef: name: aspc-test-kube-secret key: kube-secret-key volumeMounts: - name: ascp-test-secret-volume mountPath: /mnt/ascp-secret readOnly: true restartPolicy: Never serviceAccountName: ascp-test-serviceaccount volumes: - name: ascp-test-secret-volume csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: aspc-test-secret-class
Deploy, and check:
[simterm]
$ kk exec -ti ascp-test-pod -- printenv | grep SECRET SECRET=secretLine
[/simterm]
Or:
[simterm]
$ kk exec -ti ascp-test-pod -- bash bash-4.2# echo $SECRET secretLine
[/simterm]
During this, we have to attach the volumes
and volumeMounts
, as it was done for mounting Secrets as a file.
Creating a SecretProviderClass from JSON
If a data in Secrets Manager and Parameter Store is stored in JSON, then we should use the jmesPath
in our SecretProviderClass.
Let’s create another secret in Secrets Manager from JSON:
[simterm]
$ aws secretsmanager create-secret --name ascp-secret-test-json --secret-string '{"username":"admin", "password":"foobar"}' { "ARN": "arn:aws:secretsmanager:us-east-1:492***148:secret:ascp-secret-test-json-iOtcBf", "Name": "ascp-secret-test-json", "VersionId": "32666608-6416-46cf-8b93-cf090eef1bc5" }
[/simterm]
Check it:
Update our SecretProviderClass:
--- apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: aspc-test-secret-class spec: provider: aws parameters: objects: | - objectName: "ascp-secret-test-string" objectType: "secretsmanager" - objectName: "ascp-ssm-test-param" objectType: "ssmparameter" - objectName: "ascp-secret-test-json" objectType: "secretsmanager" jmesPath: - path: "username" objectAlias: "ascp-test-username" - path: "password" objectAlias: "ascp-test-password" secretObjects: - secretName: aspc-test-kube-secret type: Opaque data: - objectName: ascp-secret-test-string key: kube-secret-key - secretName: aspc-test-kube-secret-json type: Opaque data: - objectName: ascp-test-username key: kube-secret-user - objectName: ascp-test-password key: kube-secret-pass
Here:
- in the
parameters.objects.objectName: "ascp-secret-test-json"
usejmesPath
, which parses our secret and receives the values of two fields –username
andpassword
, for which it creates twoobjectAlias
– ascp-test-username and ascp-test-password - in the
secretObjects.secretName: aspc-test-kube-secret-json
we adddata
with twoobjectName
, in which we useobjectAlias
from theparameters
Update our Kubernetes Pod – add two secretKeyRef
with the kube-secret-user
and kube-secret-user
keys from the aspc-test-kube-secret-json
Secret:
--- apiVersion: v1 kind: Pod metadata: name: ascp-test-pod spec: containers: - name: my-aws-cli image: amazon/aws-cli:latest command: ['sleep', '36000'] env: - name: SECRET valueFrom: secretKeyRef: name: aspc-test-kube-secret key: kube-secret-key - name: USER valueFrom: secretKeyRef: name: aspc-test-kube-secret-json key: kube-secret-user - name: PASS valueFrom: secretKeyRef: name: aspc-test-kube-secret-json key: kube-secret-pass volumeMounts: - name: ascp-test-secret-volume mountPath: /mnt/ascp-secret readOnly: true restartPolicy: Never serviceAccountName: ascp-test-serviceaccount volumes: - name: ascp-test-secret-volume csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: aspc-test-secret-class
Deploy and check Kubernetes Secrets:
[simterm]
$ kk get secret NAME TYPE DATA AGE aspc-test-kube-secret Opaque 1 2s aspc-test-kube-secret-json Opaque 2 2s
[/simterm]
And a value from the aspc-test-kube-secret-json
:
[simterm]
$ kk get secret aspc-test-kube-secret-json -o yaml apiVersion: v1 data: kube-secret-pass: Zm9vYmFy kube-secret-user: YWRtaW4= ...
[/simterm]
And environment variables in the Pod:
[simterm]
$ kk exec -ti ascp-test-pod -- printenv | grep 'SECRET\|USER\|PASS' SECRET=secretLine USER=admin PASS=foobar
[/simterm]
Done.