Kubernetes: ServiceAccounts, JWT-tokens, authentication, and RBAC authorization

By | 11/22/2020
0 Comment
 

For the authentification and authorization, Kubernetes has such notions as User Accounts and Service Accounts.

User Accounts – common user profiles used to access a cluster from the outside, while Service Accounts are used to grant access from inside of the cluster.

ServiceAccounts are intended to provide an identity for a Kubernetes Pod to be used by its container to authenticate and authorize them when performing API-requests to the Kubernetes API-server.

Default ServiceAccount

Every Kubernetes Namespace has its own default ServiceAccount (SA) which is created when creating a namespace.

Let’s check the default namespace:

[simterm]

$ kubectl --namespace default get serviceaccount
NAME                   SECRETS   AGE
default                1         176d

[/simterm]

For each ServiceAccount a token is generated and stored as a Kubernetes Secret.

Check the default SA:

[simterm]

$ kubectl --namespace default get serviceaccount default -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2020-05-25T12:04:49Z"
  name: default
  namespace: default
  resourceVersion: "296"
  selfLink: /api/v1/namespaces/default/serviceaccounts/default
  uid: 19cc2b5f-fbc3-403e-a7c7-d62361a4038a
secrets:
- name: default-token-292g9

[/simterm]

Here is the token for this SA – the default-token-292g9 Secret:

[simterm]

...
secrets: 
- name: default-token-292g9

[/simterm]

default token

Now, check the Secret’s content:

[simterm]

$ kubectl get secret default-token-292g9 -o yaml
apiVersion: v1
data:
  ca.crt: LS0...sdA==
  token: ZXl...TWc=
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: default
    kubernetes.io/service-account.uid: 19cc2b5f-fbc3-403e-a7c7-d62361a4038a
  creationTimestamp: "2020-05-25T12:04:49Z"
  name: default-token-292g9
  namespace: default
  resourceVersion: "294"
  selfLink: /api/v1/namespaces/default/secrets/default-token-292g9
  uid: 07a46645-0083-45a0-a640-6e6a78ebd9b1
type: kubernetes.io/service-account-token

[/simterm]

At first, its type is the kubernetes.io/service-account-token.

Another interesting part here is the data which keeps two records – ca.cert и token.

If a token is not from the default namespace – there will be a third field specifying a namespace to which this token belongs.

Theca.cert is signed by the cluster’s master key so the cluster is playing the Certificate Authority role, and allows a pod or an application to verify the API-server.

And now, let’s go to investigate the tokenpart.

JWT token

To make it easier to work from the terminal – save the data.token value to a variable:

[simterm]

$ token="ZXl...TWc="

[/simterm]

Use the base64 get its content:

[simterm]

$ echo $token | base64 -d
eyJ[...]iJ9.eyJ[...]ifQ.g5I[...]3Mg

[/simterm]

Here I’ve removed some data with the […], but we can see that the value is divided into three parts with dots:

  • the header – describes how the token was signed
  • the payload – actual data of the token, such as expiration date, who issued it, etc see the RFC-7519
  • the signature – is used to verify that the token wasn’t modified and can be used to validate the sender

See the documentation>>>.

To check the token’s content we can use the jwtutility or on the jwt.io website.

In our case, the payload section has the following lines:

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "default-token-s8m4t",
  "kubernetes.io/serviceaccount/service-account.name": "default",
  "kubernetes.io/serviceaccount/service-account.uid": "b4514006-4c9a-4c30-92c8-1cc1c058b31c",
  "sub": "system:serviceaccount:default:default"
}

here in the sub filed we can see the ServiceAccount name, i.e. – who is presenting this token to the Kubernetes API-server so the server will know from who this token came.

Okay, but what about a password? In the sub there is a “login” – but where is his “password”?

And here is the third part is playing  – the signature.

JWT token and authentification

I wasn’t able to see these details in any from the googled materials, see the Useful links section of this post, although as for me – this is the most interesting part of the scheme.

Let’s go back to the first section of the token – the header, which in our case has the RS256 algorithm type defined i.e. RSA (Rivest-Shamir-Adleman) – the asymmetric algorithm with private and public keys and uses SHA-256 algorithm for the signature.

Let’s check our token on the jwt.io:

Invalid Signature – as we not provided the private and public keys to verify the token.

Because the masters’ private key on AWS Elastic Kubernetes Service is stored on the ConrolPlane nodes and we can’t access them – let’s use minikube for the testing.

Run a local cluster:

[simterm]

$ minikube start

[/simterm]

In its default namespace we can see already existing token:

[simterm]

$ kubectl get secrets                                                                                                                                                                                
NAME                  TYPE                                  DATA   AGE                                                                                                                                                                       
default-token-s8m4t   kubernetes.io/service-account-token   3      2m44s

[/simterm]

Grab the tokenfield and decode it with base64:

[simterm]

$ kubectl get secrets -o jsonpath='{.items[0].data.token}' | base64 -d
eyJhbGciO[...]61O_LxbM_-tiLjyjeCZw

[/simterm]

Go back to the jwt.io, paste the string received above:

Still Invalid Signature – but go to your minikube and take its public certificate – the ~/.minikube/ca.crt file:

[simterm]

$ cat ~/.minikube/ca.crt 
-----BEGIN CERTIFICATE-----
MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
...
0g+FhVM92T+yV38vYLO/HaKeiOzIcgHHkAoLJZd/K/Mu7crwIuGlcCVhrjcHoa3p
Md34ZTeqxA4J3w==
-----END CERTIFICATE-----

[/simterm]

Paster it to the Public Key or Certificate field.

Find the private key of the minikube cluster – actually, it is also used to sing the ca.crt and tokens, the ~/.minikube/ca.key file:

[simterm]

$ cat ~/.minikube/ca.key 
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAtDRDag2D7UBaBmWQwTKVLjuKTuat4eD/oThRgfi5bcCnwooG
...
xnL96EHthflb3NaS4GKuJYzNAPhfOdMw96Ce8KtNYpMYjRhNF9TN
-----END RSA PRIVATE KEY-----

[/simterm]

Paste it to the  Private Key field:

Signature Verified – yup, it works! The authenticity of the bearer of the token is verified.

So, going back to the ServiceAccounts:

  • for a ServiceAccount a token is created which keep the SA name
  • the token is signed by the master key of the Kubernetes cluster
  • a pod make a request to the API server using this token to authenticate him
  • the API server validates the token by using its public key and verify that the token wasn’t modified and is relly issued by this Kubernetes clutserм

Now, let’s go to see in practice how this is working and how Kubernetes RBAC is used here.

ServiceAccounts, and RBAC

For each Pod that has no ServiceAccount specified the default ServiceAccount is attached and its default token is mounted.

Go back to our EKS cluster and run a Pod:

[simterm]

$ kubectl run -i --tty --rm ca-test-pod --image=radial/busyboxplus:curl
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$

[/simterm]

Check itvolumeMounts, serviceAccount, and volumes:

[simterm]

$ kubectl get pod ca-test-pod-5c96c78d7f-wqlsq -o yaml
apiVersion: v1
kind: Pod
...
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-292g9
      readOnly: true
...
  serviceAccount: default
  serviceAccountName: default
...
  volumes:
  - name: default-token-292g9
    secret:
      defaultMode: 420
      secretName: default-token-292g

[/simterm]

Inside of the pod check the /var/run/secrets/kubernetes.io/serviceaccount directory content:

[simterm]

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ ls -1 /var/run/secrets/kubernetes.io/serviceaccount
ca.crt
namespace
token

[/simterm]

And recall the content of the data section of the default-token-292g9 Secret:

[simterm]

$ kubectl get secret default-token-292g9 -o yaml
apiVersion: v1
data:
  ca.crt: LS0t[...]
  namespace: ZGVmYXVsdA==
  token: ZXlKaGJ
...

[/simterm]

Try to perform a request to the API-server without authentification – use the special Service kubernetes, add the -k or --insecure to the curl to skip server’s certificate validation

[simterm]

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ curl -k https://kubernetes
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

[/simterm]

Cool – we got the 403, Forbidden.

Now, add two variables – one with the ca.crt and with the token:

[simterm]

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ CERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

[/simterm]

And run curl again – let’s try to get a list of the pods in our namespace, this time without --insecure and with authorization by using the Authorization header:

[simterm]

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ curl --cacert $CERT -H "Authorization: Bearer $TOKEN" "https://kubernetes/api/v1/namespaces/default/pods/"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

[/simterm]

At this time, we are able to see our user – the User "system:serviceaccount:default:default", but it has no permissions to perform requests as by default all users and ServiceAccounts have no privileges (the principle of the least privileges, POLP).

RoleBindig for ServiceAccount

To give our SericeAccount permissions we need to create a RoleBinding or ClusterRoleBinding as for normal users.

Create a RoleBinding mapping to the default ClusterRole view, see User-facing roles:

[simterm]

$ kubectl create rolebinding ca-test-view --clusterrole=view --serviceaccount=default:default
rolebinding.rbac.authorization.k8s.io/ca-test-view created

[/simterm]

And run curl again:

[simterm]

[ root@ca-test-pod-5c96c78d7f-wqlsq:/ ]$ curl --cacert $CERT -H "Authorization: Bearer $TOKEN" "https://kubernetes/api/v1/namespaces/default/pods/"
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/default/pods/",
    "resourceVersion": "66892356"
  },
  "items": [
    {
      "metadata": {
        "name": "ca-test-pod-5c96c78d7f-wqlsq",
        "generateName": "ca-test-pod-5c96c78d7f-",
        "namespace": "default",
        "selfLink": "/api/v1/namespaces/default/pods/ca-test-pod-5c96c78d7f-wqlsq",
        "uid": "f0d77cfe-38ab-48e9-aaf3-f344f1d343f3",
        "resourceVersion": "66888089",
        "creationTimestamp": "2020-11-17T16:08:09Z",
        "labels": {
          "pod-template-hash": "5c96c78d7f",
          "run": "ca-test-pod"
        },
        ...
        "qosClass": "BestEffort"
      }
    }
  ]

[/simterm]

ServiceAccounts and security

Remember, that having access to Secrets and ServiceAccounts any pod can have any token attached and thus can be able to perform actions allowed by such a token.

For example, by using the ServiceAccount of the ExternalDNS – such a pod can make a mess in our AWS Route53.

That’s why it is important to divide access to resources by using RBAC rules and roles for users, for example by allowing access to resources from only one namespace.

Useful links