AWS Elastic Kubernetes Service: RBAC Authorization via AWS IAM and RBAC Groups

We have two new projects in the Elastic Kubernetes Service (см. AWS Elastic Kubernetes Service: a cluster creation automation, part 1 – CloudFormation), each project lives in its own separate Namespace.

In addition, there are two users, developers, who need to be given access to these two Namespaces, but only to Pods in them and only for certain read-only operations.

In order to implement common access control for these users, we need some kind of group to which we can add them and then use it to check access rules.

To better understand the idea described below – see posts Kubernetes: part 4 – AWS EKS authentication, aws-iam-authenticator, and AWS IAM and Kubernetes: part 5 — RBAC authorization with a Role and RoleBinding example.

Note: the original post was written on 12/04/2020, so some parts might be slightly outdated. But the general idea and some technical details still worth knowing.

What have we have now?

We have the AWS IAM – its users, groups, roles and policies.

And we have Kubernetes RBAC with its ClusterRole, RoleBinding, and the aws-auth ConfigMap,

This means that our task is divided into two:

think about a mechanism for authentication and authorization of two users in AWS IAM that can create a single object for identification (group, role)
think through the mechanism of authorization of two users in Kubernetes RBAC using this object

The Problem

What’s the non-obvious difficulty?

In how we link these two tasks, since unlike IAM Users and IAM Roles, an AWS IAM Group cannot be the subject of authentication, see the list in AWS JSON Policy Elements: Principal, and we will not be able to use the ARN of such a group of the form arn:aws:iam::111:group/group-name in our aws-auth ConfigMap to give users of that group access to the Kubernetes cluster.

But we can use IAM Roles, and via IAM Groups we can connect common IAM policies with access to the desired roles.

What can we build using it?

create an IAM Role with an IAM read-only policy on the API calls eks:*, and add a trust policy to it with users of our account – then we will use this role in the aws-auth ConfigMap
create an IAM Group with an IAM policy that allows the sts::AssumeRole call to the IAM Role created above to be executed
create an IAM User, add it to this Group so that it “inherits” the AssumeRole permission through the group Policy.

Let’s look ahead a bit – what’s next for Kubernetes?

There our aws-auth ConfigMap will “map” our IAM Role to an RBAC group in the cluster, and our users will configure their AWS CLI to execute the AssumeRole of the actual Role arn:aws:iam::534***385:role/iam-bttrm-web-ro-role – then Kubernetes will receive an identifier common to both of there users, and they will be able to perform operations defined in their common RBAC Role and only in the allowed namespaces.

About names

So we have two users – let’s combine them into a conditional group “web” which will be used in IAM role and group names.

Plus we have two projects in Kubernetes with separate namespaces – project1 and project2, which we will also combine into the conditional group “web“, but we will use this group only in the names of RBAC resources in Kubernetes to make them more visible.

In addition, we will add the prefix iam- to AWS IAM resource names, and rbac- to RBAC resource names.

Hence, we will have names of the species:

iam-bttrm-web-ro-role: here iam says this is an object from AWS IAM, bttrm is the common name of my project, web is the “domain” for our two new web projects
rbac-bttrm-web-ro-role-binding: rbac is the object linked to Kubernetes RBAC, bttrm is the common name of my project, web is the “domain” for our two web projects

Contents

AWS IAM

To visualize what exactly we are doing and how it all will work with each other, we can sketch a scheme like this:

Here:

A user needs to access a Role by executing an AssumeRole request in AWS IAM:
1. AWS IAM performs authentication of the user
2. if authentication is passed – AWS IAM starts authorization of the user: checks the access policies connected to it
3. finds a policy that allows AssumeRole for the desired role and allows the user to access that role
the user accesses the IAM Role:
1. AWS IAM validates the Trust relations of this Role
2. makes sure that the user came from the same account as our Role and allows access
the user accesses the AWS Elastic Kubernetes Service, but already using the IAM Role identifier (token)

This and the following diagram were made in the cloudcraft.co, but in the workflow such ones are simply drawn by hand in a large notepad.
I don’t know why no one in any of the guides I’ve seen has shown all these connections in the form of such simple and visual diagrams – without them, it’s much more difficult to understand all this, especially in the part Kubernetes RBAC.

We’ll leave it here for now, we can come back as we go.

Creating An IAM Role

Create a new role – the key, in fact, element of the entire design.

Use ЕС2:

On the next page click Create policy, in the new tab in the policy creation window switch to JSON, enter the policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster",
                "eks:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}

In it, we allow the eks:DescribeCluster and eks:ListClusters API calls to be executed across all regions of our entire account – these calls are used by the AWS CLI during the execution of aws eks update-kubeconfig, which will be executed by our new users to configure their local kubectl.

Save it with the name iam-bttrm-eks-ro-policy.

Go back to the previous tab with the role creation, click on “Update” on the right, and add the created policy to the role:

Skip the Tags, save the Role with the name iam-bttrm-web-ro-ro-role:

Remember its Role ARN:

IAM Role Trust relationships

Although here>>> the answers claim that Trust relations within a single account don’t need to be created – but it didn’t work for me without it, plus it’s mentioned in the documentation here t>>>>.

Switch to the Trust relationships tab, click Edit trust relationship, in Principal specify “AWS”: “arn:aws:iam::534***385:root” – here root includes all users (or rather, all authentication objects) of our account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::534***385:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Note: actually, from the security point of view it’s a bad idea to give such a wide access, via the root entity

But it doesn’t mean that any subject authenticated in our account will be able to use (assume) this role, because this subject must still have the corresponding rights to call this particular Role – and for our future users we are implementing this through IAM Group.

Creating an IAM Policy

Before we create an IAM Group – let’s add the actual policy that will allow the execution of the API call sts:AssumeRole to the Role we created, and then this Policy will be connected to the created Group, through which it will be “inherited” by future users of the group.

Later, to take away the IAM-user’s access to an Elastic Kubernetes Service cluster, it will be enough to disconnect him from the IAM Group.

Create a policy:

In the Resource, specify the ARN of the Policy we created above:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::534***385:role/iam-bttrm-web-ro-role"
    }
}

Save it, for example with the name iam-bttrm-allow-assume-web-ro-role-policy:

Creating an IAM Group

Go to Groups, create a new one, name it iam-bttrm-web-ro-group, connect the previously created Policy to it:

Creating an IAM User

Create a user with Programmatic access, add it to the iam-bttrm-web-ro-group, were we already have the am-bttrm-allow-assume-web-ro-role-policy connected:

Configure the local AWS CLI profile to a new user iam-bttrm-web-user-1:

$ aws configure --profile iam-bttrm-web-user-1
AWS Access Key ID [None]: AKI***O4Z
AWS Secret Access Key [None]: FoS***kft
Default region name [None]: us-east-2
Default output format [None]: json

Check it:

$ aws --profile iam-bttrm-web-user-1 sts get-caller-identity
{
    "UserId": "AID***DUH",
    "Account": "534***385",
    "Arn": "arn:aws:iam::534***385:user/iam-bttrm-web-user-1"
}

Trying to access EC2 – should get a rejection:

$ aws --profile iam-bttrm-web-user-1 ec2 describe-instances

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.

Okay – no one gave us permission to make an API request DescribeInstances, and we were not authorized (although we were authenticated as user iam-bttrm-web-user-1).

Try EKS operations:

$ aws --profile iam-bttrm-web-user-1 ec2 describe-instances

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.

Great – we authenticated again as iam-bttrm-web-user-1 – but again failed authorization, this time to execute the eks:ListClusters API call.

And now, let’s do this by using AssumeRole:

authenticate as iam-bttrm-web-user-1
check sts:AssumeRole – IAM will check what policies are attached to us and find the iam-bttrm-allow-assume-web-ro-role-policy policy
we will temporarily gain rights to perform operations allowed for the iam-bttrm-web-ro-role IAM Role with the eks:ListClusters permissions

Let’s try it.

Update the local ~/.aws/config, and add a second profile, iam-bttrm-web-user-1-eks, to the iam-bttrm-web-user-1-eks profile:

...
[profile iam-bttrm-web-user-1]
region = us-east-2
output = json

[profile iam-bttrm-web-user-1-eks]
role_arn = arn:aws:iam::534***385:role/iam-bttrm-web-ro-role
source_profile = iam-bttrm-web-user-1
region = us-east-2

In which we use the arn:aws:iam::534***385:role/iam-bttrm-web-ro-role role, but authenticate as iam-bttrm-web-user-1 via the source_profile. (see Using an IAM Role in the AWS CLI).

Try it – repeat under a common profile:

$ aws  --profile iam-bttrm-web-user-1 eks list-clusters --output text

An error occurred (AccessDeniedException) [...]

And under the second, with the AssumeRole:

$ aws  --profile iam-bttrm-web-user-1-eks eks list-clusters --output text
CLUSTERS        bttrm-eks-prod-0
CLUSTERS        eksctl-bttrm-eks-production-1
CLUSTERS        bttrm-eks-dev-0

Everything works, we’re done with that – we have a group using which users get to execute eks:ListClusters.

Using the same role (but their ACCESS and SECRET keys for authentication) users will configure their kubectl to access the Pods. But we will restrict access to Namespaces and Pods via Kubernetes RBAC.

Kubernetes RBAC

So, we need to have some group through permissions to which we can grant access to pods in two different namespaces, and this group must be associated with an AWS IAM Group.

Since we can’t use AWS IAM Group ARN – we do a “dirty hack” in the form of using a common role for different users of the group, by which we will identify them for authorization in Kubernetes RBAC.

Next, we need to create:

a ClusterRole – role that will authorize operations with Pods in all Namespaces
two RoleBinding in two Namespaces, one in each, which will bind the RBAC group and the ClusterRole in this Namespace, thus limiting the impact of this ClusterRole for the RBAC group to the boundaries of the Namespace in which this RoleBinding is created
and update the aws-auth ConfigMap to link the IAM Role and the RBAC group

As a result, we should have the following scheme:

Creating an RBAC ClusterRole

Describe a ClusterRole named rbac-bttrm-pods-ro-cluster-role:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rbac-bttrm-pods-ro-cluster-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Create it:

$ kubectl apply -f rbac-bttrm-pods-ro-cluster-role.yml 
clusterrole.rbac.authorization.k8s.io/rbac-bttrm-pods-ro-cluster-role created

Creating an RBAC RoleBinding

Create two RoleBinding – identical but in different namespace – bttrm-web-proj-1-ns and bttrm-web-proj-2-ns:

--
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rbac-bttrm-web-proj-1-ro-role-binding
  namespace: bttrm-web-proj-1-ns
subjects:
- kind: Group
  name: rbac-bttrm-web-ro-group
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: rbac-bttrm-pods-ro-cluster-role
  apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rbac-bttrm-web-proj-2-ro-role-binding
  namespace: bttrm-web-proj-2-ns
subjects:
- kind: Group
  name: rbac-bttrm-web-ro-group
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: rbac-bttrm-pods-ro-cluster-role
  apiGroup: rbac.authorization.k8s.io

Create the namespaces themselves:

$ kubectl create ns bttrm-web-proj-1-ns
namespace/bttrm-web-proj-1-ns created
$ kubectl create ns bttrm-web-proj-2-ns
namespace/bttrm-web-proj-2-ns created

Create NGINX Pods in them:

$ kubectl -n bttrm-web-proj-1-ns run nginx --image=nginx 
$ kubectl -n bttrm-web-proj-2-ns run nginx --image=nginx

And create the Bindings:

$ kubectl apply -f rbac-bttrm-web-ro-role-binding.yml 
rolebinding.rbac.authorization.k8s.io/rbac-bttrm-web-proj-1-ro-role-binding created
rolebinding.rbac.authorization.k8s.io/rbac-bttrm-web-proj-2-ro-role-binding created

Updating aws-auth `ConfigMap`

Now we need to update the aws-auth ConfigMap, which would link our IAM Group in AWS (and essentially a shared IAM Role) to the rbac-bttrm-web-ro-group “virtual” group we “created” in bindings.

Edit it:

$ kubectl -n kube-system edit cm aws-auth

Add the Role ARN to the mapGroups:

...
    - groups:
      - rbac-bttrm-web-ro-group
      rolearn: arn:aws:iam::534***385:role/iam-bttrm-web-ro-role
      username: iam-bttrm-web-ro-role 
...

Save, exit.

`kubecl` config and verification

Now for our kubectl, let’s set up an access context to our Dev cluster using the AWS CLI profile iam-bttrm-web-user-1-eks, which runs AssumeRole arn:aws:iam::534***385:role/iam-bttrm-web-ro-role, with the IAM user iam-bttrm-web-user-1 used for authentication:

$ aws --profile iam-bttrm-web-user-1-eks eks update-kubeconfig --name bttrm-eks-dev-0
Updated context arn:aws:eks:us-east-2:534***385:cluster/bttrm-eks-dev-0 in /home/setevoy/.kube/config

Try just get pod in the default namespace:

$ kubectl get pod
Error from server (Forbidden): pods is forbidden: User "iam-bttrm-web-ro-role" cannot list resource "pods" in API group "" in the namespace "default"

And repeat in the bttrm-web-proj-1-ns:

$ kubectl get pod -n bttrm-web-proj-1-ns
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7bb7cd8db5-l9jtm   1/1     Running   0          2m24s

Here’s our NGINX.

Repeat for the namespace of the second project – bttrm-web-proj-2-ns:

$ kubectl get pod -n bttrm-web-proj-2-ns
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7bb7cd8db5-9s6zj   1/1     Running   0          2m31s

You can try accessing other resources that we didn’t give access to, such as WorkerNodes:

$ kubectl get node
Error from server (Forbidden): nodes is forbidden: User "iam-bttrm-web-ro-role" cannot list resource "nodes" in API group "" at the cluster scope

Or use kubectl auth can-i:

$ kubectl auth can-i get node
Warning: resource 'nodes' is not namespace scoped
no

$ kubectl auth can-i get pod
no

$ kubectl auth can-i get pod -n bttrm-web-proj-1-ns
yes

$ kubectl auth can-i create pod -n bttrm-web-proj-1-ns
no

Done.

M	T	W	T	F	S	S
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30