So, as a follow-up to the Helm: Kubernetes package manager – an overview, getting started post – let’s discuss about sensitive data in our Helm charts.
What I want is to store a chart files in a repository, but even if such a repo will be a private Github repo – I still don’t want to store passwords in a plaintext way there.
The solution here might be to use a well-known helm-secrets
plugin. The plugin itself is just a collection if bash-scripts using Mozilla SOPS (SOPS – Secrets OPerationS), and is a wrapper around helm secrets command.
It allows us to encrypt strings in a specified file using GPG/AWS KMS/GCP KMS keys and decrypt such a data on the fly to embed it as a usual value in the same way as we doing it using the common values.yaml
file.
So, in this post, we will install the plugin on Arch Linux, will create an AWS KMS key, then will see how to encrypt/decrypt data in a Helm chart, and then will use it in a Jenkins job which is used in my job’s project to deploy a real working application.
Did a nice typo during writing this post, btw:
Contents
The helm-secrets
plugin install
Actually, the installation must be done just by typing “helm plugin install
“, but its installation script a bit awkward and may not work properly on some operating systems.
helm-secrets
& sops
on Arch Linux
For example, on my Arch Linux first I got permissions issue:
And then again permissions error, and anyway I have no willingness to install it to the /root
directory using the sudo
:
So – just install the sops
package from AUR:
Or, if you are macOS user – use the brew
:
Then repeat the helm plugin install https://github.com/futuresimple/helm-secrets
command and check the documentation – let’s take a look at what it installed for us.
It has to use the $XDG_DATA_HOME/helm/plugins
directory, and $XDG_DATA_HOME
has to be pointed to the $HOME/.local/share
location:
And the plugin’s directory content is the following:
Here are the bash-scripts mentioned above and in the Moving parts of project documentation.
Also, we can inspect the plugin.yaml
file to see which actions will be performed by the plugin:
command: "$HELM_PLUGIN_DIR/secrets.sh"
– well, yes – the secrets.sh
is the main working script of the plugin.
You can also check it’s content – but it’s really big enough:
Encryption configuration
To find which keys are used for encryption and decryption process – the.sops.yaml
file will be used, which is stored in the root of your chart’s directory.
Actually, this file is used by the SOPS utility, so you can find it’s configuration examples in the Using .sops.yaml conf to select KMS/PGP for new files doc.
Now we can go ahead and create an AWS KMS key which will be used to protect our data.
AWS KMS – a key creation
If you didn’t use the AWS KMS service before – check its docs here – https://aws.amazon.com/ru/kms/getting-started.
Go to the AWS Console and create a new, symmetric, key (see the Using symmetric and asymmetric keys):
Set its Administrator:
Now, we need to add it’s users.
As I’ll use the key from my Jenkins instance which is running on an AWS EC2, and this EC2 instance has the AWS EC2 Instance Profile attached to authenticate for the AWS services – then I need to add its IAM Role as a User for the key.
Go to the Jenkins EC2 and find its IAM Role (at the bottom of the screenshot below):
Attach it to the Users of the key we are creating:
Confirm the policy generated:
By the way – pay attention here:
... "Principal": { "AWS": "arn:aws:iam::534***385:root" }, "Action": "kms:*", "Resource": "*" ...
The “arn:aws:iam::534***385:root” ARN refers to the all the users of the AWS account, as I remember from the AWS Elastic Kubernetes Service: RBAC-авторизация через AWS IAM и RBAC группы post (in Russian only for now), so keep it in mind.
Press the Finish button and the key is ready:
The .sops.yaml
config
Now, go to your chart’s directory and create the .sops.yaml
file to configure the key used for our secrets.
Let’s use the only one default rule here to be applied for any secrets file with the AWS KMS key we created above:
--- creation_rules: - kms: 'arn:aws:kms:eu-west-1:534***385:key/620b89fe-***-25b435611e8b'
A secrets file
Next, create a new file named secrets.yaml
.
Here is a small trap – the script will look for the secrets.*.yaml files by default, see the Usage and examples, and you can’t just use a name like “my-passwords.yaml” at least without some additional configuration:
By convention, files containing secrets are named secrets.yaml, or anything beginning with “secrets.” and ending with “.yaml”. E.g. secrets.test.yaml and secrets.prod.yaml.
So, for example, I have the following strings in my existing values.yaml
– and here is a “password” key with a plaintext value “pass“:
... image: registry: "docker.io" username: "bttrm" password: "pass" repository: "bttrm" name: "bttrm-apps" tag: "120" ...
Cut the image.password
from the values.yaml
and move it to the secrets.yaml
:
image: password: "pass"
Or, to make this example more clear and simple – let’s also add a new key and value to this secrets.yaml
:
test: secret: testecret
And then create a manifest for Kubernetes Secrets, in my case all the Secrets are described in the bttrm-apps-backend/templates/bttrm-apps-secrets.yaml
file, where bttrm-apps-backend is the chart’s directory:
--- apiVersion: v1 kind: Secret metadata: name: test-secret type: Opaque stringData: example-secret: {{ .Values.test.secret }}
As you can see – in the stringData
field we are using the .Values
as usually for the common values.yaml
file.
helm secrets
– AccessDeniedException
Now, we can encrypt the file using the helm secrets enc
command:
Ah, okay.
The issue is obvious enough – the SOPS is trying to access the AWS KMS key specified in the .sops.yaml
config using the default AWS CLI profile from the ~/.aws/credentials
, where I have my personal AWS account configured, while the key is located in my work account.
So, in this case, you can specify a profile using the aws_profile
option in the .sops.yaml
:
--- creation_rules: - kms: 'arn:aws:kms:eu-west-1:534***385:key/620b89fe-***-25b435611e8b' aws_profile: 'arseniy'
But then the same account will be used from a Jenkins job.
Another way is to use the $AWS_PROFILE
environment variable:
Repeat the secrets.yaml
file encryption in the bttrm-apps-backend directory:
Check its content now:
image: password: ENC[AES256_GCM,data:1t7 .. PyY,iv:9mzTyB ... 9KG7+Hg=,tag:6KJ ... gvA==,type:str] test: secret: ENC[AES256_GCM,data:l4c ... vCy,iv:9riw ... 0fvQ=,tag:G3p ... oSw==,type:str] sops: kms: - arn: arn:aws:kms:eu-west-1:534***385:key/620b89fe- ... -25b435611e8b created_at: '2020-05-15T06:33:44Z' enc: AQICAHj9F0HBsgu ... kb6GOxJkiOMZSSxOtA== ...
As you can see, the values are encrypted now, and then there is information for the SOPS about configuration applied during the encryption process.
You can see the initial data using the secrets view
command:
And edit the file with the secrets edit
:
And decrypt with the secrets dec
command – this will create a new secrets.yaml.dec
file near the original secrets.yaml
.
Test deploy
To deploy the chart with the encrypted data – call helm
with the install
or upgrade
as usual, or as in my case with the Jenkins job (will speak about it a bit later) – upgrade --install
, but now add the secrets
command – helm
will execute the secrets.sh
script which will perform all necessary operations.
In the example below I’m installing the bttrm-apps-backend/ chart as a bttrm-apps-backend release using the values.yaml
and secrets.yaml
files as the source of values (actually – you can skip the default values.yaml
file and specify the only secrets.yaml
as an additional file):
Here might be interesting to check the processes started during the encryption process after you executed the helm secrets upgrade
:
E.g:
- we ran the
helm secrets upgrade
- it called the the
bash /home/setevoy/.local/share/helm/plugins/helm-secrets/secrets.sh upgrade
- which executed the
helm upgrade
Now – check the secret in the Kubernetes:
And value from the dGVzdGVjcmV0 base64-encoded string is:
Cool – “It works!” (c)
Go to the Jenkins job.
Jenkins
Now, I need to embed the things above to a Jenkinsfile, which I wrote in the Helm: пошаговое создание чарта и деплоймента из Jenkins post (also still in Russian, unfortunately) – the script will build a Docker image and will deploy the bttrm-apps-backend chart as bttrm-apps-backend release to a Dev-cluster.
If delete the release now:
And run the Jenkins job it obviously will fail as helm
will not be able to find the data which we moved from the values.yaml
into the secrets.yaml
file:
So to make it great again working again need to add the SOPS tool and the helm-secrets
plugin to my Docker image used for the build, and then update the stage(“Helm install”) in the script to add the secrets
command and -f secrets.yaml
.
Jenkins Docker build-image
Currently, my Docker image is built using the next Dockerfile:
FROM bitnami/minideb:stretch RUN apt update && install_packages ca-certificates wget RUN install_packages curl python-pip python-setuptools jq git RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl RUN chmod +x ./kubectl RUN mv ./kubectl /usr/local/bin/kubectl WORKDIR /tmp RUN curl --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp RUN mv /tmp/eksctl /usr/local/bin RUN pip install ansible boto3 awscli WORKDIR /tmp RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 RUN /bin/bash get_helm.sh USER root
Add the plugin install – there is the bitnami/minideb:stretch
image used, so I hope that SOPS will be installed without any issues as it was with my Arch Linux:
... RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 RUN /bin/bash get_helm.sh RUN helm plugin install https://github.com/futuresimple/helm-secrets USER root
Try to build:
Well – no 🙂
Okay – we can install it using PIP:
... RUN mv /tmp/eksctl /usr/local/bin RUN pip install ansible boto3 awscli sops WORKDIR /tmp ...
Although during build Helm told that:
But let’s see if this will work in a Jenkins build.
Jenkinsfile and deploy job
The last step is to update the pipeline’s Jenkinsfile – add the secrets
and -f bttrm-apps-backend/secrets.yaml
:
... stage("Helm install") { ... sh "helm secrets upgrade --install --namespace ${AWS_EKS_NAMESPACE} --create-namespace --atomic ${APP_CHART_NAME} ${APP_CHART_NAME}-${RELEASE_VERSION}.tgz -f bttrm-apps-backend/secrets.yaml" } } } ...
Run:
And it’s ready.
Useful links
Similar posts
Also published on Medium.