So, as a follow-up to the Helm: Kubernetes package manager – an overview, getting started post – let’s discuss about sensitive data in our Helm charts.

What I want is to store a chart files in a repository, but even if such a repo will be a private Github repo – I still don’t want to store passwords in a plaintext way there.

The solution here might be to use a well-known helm-secrets plugin. The plugin itself is just a collection if bash-scripts using Mozilla SOPS (SOPS – Secrets OPerationS), and is a wrapper around helm secrets command.

It allows us to encrypt strings in a specified file using GPG/AWS KMS/GCP KMS keys and decrypt such a data on the fly to embed it as a usual value in the same way as we doing it using the common  values.yaml file.

So, in this post, we will install the plugin on Arch Linux, will create an AWS KMS key, then will see how to encrypt/decrypt data in a Helm chart, and then will use it in a Jenkins job which is used in my job’s project to deploy a real working application.

Did a nice typo during writing this post, btw:

The helm-secrets plugin install

Actually, the installation must be done just by typing “helm plugin install“, but its installation script a bit awkward and may not work properly on some operating systems.

helm-secrets & sops on Arch Linux

For example, on my Arch Linux first I got permissions issue:

And then again permissions error, and anyway I have no willingness to install it to the /root directory using the sudo:

So – just install the sops package from AUR:

Or, if you are macOS user – use the brew:

Then repeat the helm plugin install https://github.com/futuresimple/helm-secrets command and check the documentation – let’s take a look at what it installed for us.

It has to use the $XDG_DATA_HOME/helm/plugins directory, and $XDG_DATA_HOME has to be pointed to the $HOME/.local/share location:

And the plugin’s directory content is the following:

Here are the bash-scripts mentioned above and in the Moving parts of project documentation.

Also, we can inspect the  plugin.yaml file to see which actions will be performed by the plugin:

command: "$HELM_PLUGIN_DIR/secrets.sh" – well, yes – the secrets.sh is the main working script of the plugin.

You can also check it’s content – but it’s really big enough:

Encryption configuration

To find which keys are used for encryption and decryption process – the.sops.yaml file will be used, which is stored in the root of your chart’s directory.

Actually, this file is used by the SOPS utility, so you can find it’s configuration examples in the Using .sops.yaml conf to select KMS/PGP for new files doc.

Now we can go ahead and create an AWS KMS key which will be used to protect our data.

AWS KMS – a key creation

If you didn’t use the AWS KMS service before – check its docs here – https://aws.amazon.com/ru/kms/getting-started.

Go to the AWS Console and create a new, symmetric, key (see the Using symmetric and asymmetric keys):

Set its Administrator:

Now, we need to add it’s users.

As I’ll use the key from my Jenkins instance which is running on an AWS EC2, and this EC2 instance has the AWS EC2 Instance Profile attached to authenticate for the AWS services – then I need to add its IAM Role as a User for the key.

Go to the Jenkins EC2 and find its IAM Role (at the bottom of the screenshot below):

Attach it to the Users of the key we are creating:

Confirm the policy generated:

By the way – pay attention here:

...
            "Principal": {
                "AWS": "arn:aws:iam::534***385:root"
            },
            "Action": "kms:*",
            "Resource": "*"
...

The “arn:aws:iam::534***385:root” ARN refers to the all the users of the AWS account, as I remember from the AWS Elastic Kubernetes Service: RBAC-авторизация через AWS IAM и RBAC группы post (in Russian only for now), so keep it in mind.

Press the Finish button and the key is ready:

The .sops.yaml config

Now, go to your chart’s directory and create the .sops.yaml file to configure the key used for our secrets.

Let’s use the only one default rule here to be applied for any secrets file with the AWS KMS key we created above:

---
creation_rules:
  - kms: 'arn:aws:kms:eu-west-1:534***385:key/620b89fe-***-25b435611e8b'

A secrets file

Next, create a new file named secrets.yaml.

Here is a small trap – the script will look for the secrets.*.yaml files by default, see the Usage and examples, and you can’t just use a name like “my-passwords.yaml” at least without some additional configuration:

By convention, files containing secrets are named secrets.yaml, or anything beginning with “secrets.” and ending with “.yaml”. E.g. secrets.test.yaml and secrets.prod.yaml.

So, for example, I have the following strings in my existing values.yaml – and here is a “password” key with a plaintext value “pass“:

...
image:
  registry: "docker.io"
  username: "bttrm"
  password: "pass"
  repository: "bttrm"
  name: "bttrm-apps"
  tag: "120"
...

Cut the image.password from the values.yaml and move it to the secrets.yaml:

image:
  password: "pass"

Or, to make this example more clear and simple – let’s also add a new key and value to this secrets.yaml:

test:
    secret: testecret

And then create a manifest for Kubernetes Secrets, in my case all the Secrets are described in the bttrm-apps-backend/templates/bttrm-apps-secrets.yaml file, where bttrm-apps-backend is the chart’s directory:

---
apiVersion: v1
kind: Secret
metadata:
  name: test-secret
type: Opaque
stringData:
  example-secret: {{ .Values.test.secret }}

As you can see – in the stringData field we are using the .Values as usually for the common values.yaml file.

helm secrets – AccessDeniedException

Now, we can encrypt the file using the helm secrets enc command:

Ah, okay.

The issue is obvious enough – the SOPS is trying to access the AWS KMS key specified in the .sops.yaml config using the default AWS CLI profile from the ~/.aws/credentials, where I have my personal AWS account configured, while the key is located in my work account.

So, in this case, you can specify a profile using the aws_profile option in the .sops.yaml:

---
creation_rules:
  - kms: 'arn:aws:kms:eu-west-1:534***385:key/620b89fe-***-25b435611e8b'
    aws_profile: 'arseniy'

But then the same account will be used from a Jenkins job.

Another way is to use the $AWS_PROFILE environment variable:

Repeat the secrets.yaml file encryption in the bttrm-apps-backend directory:

Check its content now:

image:
    password: ENC[AES256_GCM,data:1t7 .. PyY,iv:9mzTyB ... 9KG7+Hg=,tag:6KJ ... gvA==,type:str]
test:
    secret: ENC[AES256_GCM,data:l4c ... vCy,iv:9riw ... 0fvQ=,tag:G3p ... oSw==,type:str]
sops:
    kms:
    -   arn: arn:aws:kms:eu-west-1:534***385:key/620b89fe- ... -25b435611e8b
        created_at: '2020-05-15T06:33:44Z'
        enc: AQICAHj9F0HBsgu ... kb6GOxJkiOMZSSxOtA==
...

As you can see, the values are encrypted now, and then there is information for the SOPS about configuration applied during the encryption process.

You can see the initial data using the secrets view command:

And edit the file with the secrets edit:

And decrypt with the secrets dec command – this will create a new secrets.yaml.dec file near the original secrets.yaml.

Test deploy

To deploy the chart with the encrypted data – call helm with the install or upgrade as usual, or as in my case with the Jenkins job (will speak about it a  bit later) – upgrade --install, but now add the secrets command  – helm will execute the secrets.sh script which will perform all necessary operations.

In the example below I’m installing the bttrm-apps-backend/ chart as a bttrm-apps-backend release using the values.yaml and secrets.yaml files as the source of values (actually – you can skip the default values.yaml file and specify the only secrets.yaml as an additional file):

Here might be interesting to check the processes started during the encryption process after you executed the helm secrets upgrade:

E.g:

1. we ran the helm secrets upgrade
2. it called the the bash /home/setevoy/.local/share/helm/plugins/helm-secrets/secrets.sh upgrade
3. which executed the helm upgrade

Now – check the secret in the Kubernetes:

And value from the dGVzdGVjcmV0 base64-encoded string is:

Cool – “It works!” (c)

Go to the Jenkins job.

Jenkins

Now, I need to embed the things above to a Jenkinsfile, which I wrote in the Helm: пошаговое создание чарта и деплоймента из Jenkins post (also still in Russian, unfortunately) – the script will build a Docker image and will deploy the bttrm-apps-backend chart as bttrm-apps-backend release to a Dev-cluster.

If delete the release now:

And run the Jenkins job it obviously will fail as helm will not be able to find the data which we moved from the values.yaml into the secrets.yaml file:

So to make it great again working again need to add the SOPS tool and the helm-secrets plugin to my Docker image used for the build, and then update the stage(“Helm install”) in the script to add the secrets command and -f secrets.yaml.

Jenkins Docker build-image

Currently, my Docker image is built using the next Dockerfile:

FROM bitnami/minideb:stretch
  
RUN apt update && install_packages ca-certificates wget
RUN install_packages curl python-pip python-setuptools jq git

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

WORKDIR /tmp
RUN curl  --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
RUN mv /tmp/eksctl /usr/local/bin

RUN pip install ansible boto3 awscli

WORKDIR /tmp
RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
RUN /bin/bash get_helm.sh

USER root

Add the plugin install – there is the  bitnami/minideb:stretch image used, so I hope that SOPS will be installed without any issues as it was with my Arch Linux:

...
RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
RUN /bin/bash get_helm.sh
RUN helm plugin install https://github.com/futuresimple/helm-secrets 

USER root

Try to build:

Well – no 🙂

Okay – we can install it using PIP:

...
RUN mv /tmp/eksctl /usr/local/bin 

RUN pip install ansible boto3 awscli sops

WORKDIR /tmp
...

Although during build Helm told that:

But let’s see if this will work in a Jenkins build.

Jenkinsfile and deploy job

The last step is to update the pipeline’s Jenkinsfile – add the secrets and -f bttrm-apps-backend/secrets.yaml:

...
        stage("Helm install") {
                    ...
                    sh "helm secrets upgrade --install --namespace ${AWS_EKS_NAMESPACE} --create-namespace --atomic ${APP_CHART_NAME} ${APP_CHART_NAME}-${RELEASE_VERSION}.tgz -f bttrm-apps-backend/secrets.yaml"
                }
            }   
        }
...

Run:

And it’s ready.

Useful links

• Secret Management in Helm – Kubernetes
• Encrypting Helm Secrets
• helm-secrets

Also published on Medium.