AWS Elastic Kubernetes Service: running ALB Ingress controller

By | 04/21/2020
 

AWS ALB Ingress Controller for Kubernetes – is a Kubernetes controller which actually controls AWS Application Load Balancers (ALB) in an AWS account when an Ingress resource with the kubernetes.io/ingress.class: alb annotation is created in a Kubernetes cluster.

This Ingress resource in its turn describes an ALB Listeners configuration with SSL termination or traffic routing to the cluster’s WorkerNodes.

More in the documentation here:

Ingress controller types

AWS ALB Ingress controller supports two policy types for traffic routing –  the instance mode and the ip mode:

  • instance mode: traffic will be accepted on an ALB, then routed to a NodePort Service, then routed to pods inside of the cluster
  • ip mode: traffic will go to an ALB first, then is routed directly to pods in a cluster. The documentation states that it’s necessary to install the AWS CNI plugin for Kubernetes, but in AWS EKS it works out of the box

See more documentation here>>>.

eksctl – create a cluster

At first, let’s create a testing cluster:

[simterm]

$ eksctl create cluster --profile arseniy --region eu-north-1 --name eks-alb-testing
[ℹ]  eksctl version 0.15.0
[ℹ]  using region eu-north-1
[ℹ]  setting availability zones to [eu-north-1b eu-north-1c eu-north-1a]
[ℹ]  subnets for eu-north-1b - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for eu-north-1c - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for eu-north-1a - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-c408bab3" will use "ami-0d40170cbe0b51e62" [AmazonLinux2/1.14]
[ℹ]  using Kubernetes version 1.14
[ℹ]  creating EKS cluster "eks-alb-testing" in "eu-north-1" region with un-managed nodes
...
[ℹ]  node "ip-192-168-11-166.eu-north-1.compute.internal" is ready
[ℹ]  node "ip-192-168-35-128.eu-north-1.compute.internal" is ready
[ℹ]  kubectl command should work with "/home/setevoy/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "eks-alb-testing" in "eu-north-1" region is ready

[/simterm]

IAM OIDC provider

Documentation – https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html.

The OIDC integration is necessary for the ServiceAccount used for the Controller, see more at https://eksctl.io/usage/iamserviceaccounts.

Create it:

[simterm]

$ eksctl --profile arseniy --region=eu-north-1 utils associate-iam-oidc-provider --cluster eks-alb-testing  --approve
[ℹ]  eksctl version 0.15.0
[ℹ]  using region eu-north-1
[ℹ]  will create IAM Open ID Connect provider for cluster "eks-alb-testing" in "eu-north-1"
[✔]  created IAM Open ID Connect provider for cluster "eks-alb-testing" in "eu-north-1"

[/simterm]

Check:

[simterm]

$ aws --profile arseniy. --region=eu-north-1 eks describe-cluster --name eks-alb-testing --query "cluster.identity.oidc.issuer" --output text
https://oidc.eks.eu-north-1.amazonaws.com/id/B59F43D7826EB61861FC73EE13216CD8

[/simterm]

Or via AWS UI > Identity providers:

ALB IAM policy

The next step is to add an IAM policy that will give access for a pod with the ALB Ingress Controller in an AWS Account to make an API-calls to the AWS Core to create and configure Application Load Balancers.

It can be created using AWS CLI with the following command:

[simterm]

$ aws --profile arseniy --region=eu-north-1 iam create-policy --policy-name ALBIngressControllerIAMPolicy --policy-document https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/iam-policy.json

[/simterm]

Or can be added to a CloudFormation template using the AWS::IAM::Role resource.

In the command’s output you’ll have an ARN – “Arn”: “arn:aws:iam::534***385:policy/ALBIngressControllerIAMPolicy” – keep it, we will use it in the following step.

ALB Ingress IAM Role

See https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

1 iamserviceaccount (kube-system/alb-ingress-controller) was excluded

In the official documentation, the order is a bit different: first, it suggests to create an RBAC-role and RoleBinding, and after that – iamserviceaccount.

In my case, this led to the error during creating a new ServiceAccount “kube-system/alb-ingress-controller” – eksctl told that it is already present and skips it despite the --override-existing-serviceaccounts option:

[simterm]

[ℹ] eksctl version 0.15.0                                                                                                                        
[ℹ] using region eu-north-1                                                                                                                      
[ℹ] 1 iamserviceaccount(s) that already exist (kube-system/alb-ingress-controller) will be excluded                                              
[ℹ] combined exclude rules: kube-system/alb-ingress-controller                                                                                   
[ℹ] 1 iamserviceaccount (kube-system/alb-ingress-controller) was excluded (based on the include/exclude rules)                                   
[!] serviceaccounts that exists in Kubernetes will be excluded, use --override-existing-serviceaccounts to override                              
[ℹ] no tasks

[/simterm]

After containing AWS SUpport (love these guys – the best support overall Cloud providers, in my opinion) we’ve changed the order: first, create the IAM-role, and after that – RBAC Role and binding.

But – let’s go ahead.

Check the currently configured cluster – just in case:

[simterm]

$ kubectl config current-context
[email protected]

[/simterm]

Okay.

Now create the IAM role using the eksctl, and using the ARN of the policy created above – attach it to this role (will create an additional CloudFormation stack):

[simterm]

$ eksctl --profile arseniy --region=eu-north-1 create iamserviceaccount --name alb-ingress-controller --namespace kube-system  --override-existing-serviceaccounts --approve --cluster eks-alb-testing --attach-policy-arn arn:aws:iam::534***385:policy/ALBIngressControllerIAMPolicy
[ℹ]  eksctl version 0.15.0
[ℹ]  using region eu-north-1
[ℹ]  1 task: { 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/alb-ingress-controller", create serviceaccount "kube-system/alb-ingress-controller" } }
[ℹ]  building iamserviceaccount stack "eksctl-eks-alb-testing-addon-iamserviceaccount-kube-system-alb-ingress-controller"
[ℹ]  deploying stack "eksctl-eks-alb-testing-addon-iamserviceaccount-kube-system-alb-ingress-controller"
[ℹ]  created serviceaccount "kube-system/alb-ingress-controller"

[/simterm]

Wait for its provision:

ALB Ingress ServiceAccount

Create a ServiceAccount named alb-ingress-controller in the kube-system namespace, add ClusterRoleBinding and ClusterRole named alb-ingress-controller with access rules:

[simterm]

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/rbac-role.yaml
clusterrole.rbac.authorization.k8s.io/alb-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/alb-ingress-controller created
serviceaccount/alb-ingress-controller created

[/simterm]

Check:

[simterm]

$ kubectl -n kube-system get serviceaccounts | grep alb
alb-ingress-controller               1         32s

[/simterm]

Start ALB Ingress Controller

We can launch it using the deployment file from here – https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/alb-ingress-controller.yaml, which will spin up a pod using the docker.io/amazon/aws-alb-ingress-controller Docker image.

Clone it to your workstation:

[simterm]

$ wget https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/alb-ingress-controller.yaml

[/simterm]

Open for edit and in the block:

...
    spec:
      containers:
        - name: alb-ingress-controller
          args:
...

Add the following arguments:

  • --cluster-name=eks-alb-testing
  • --aws-vpc-id=vpc-06a182dd0e5ef1b70
  • --aws-region=eu-north-1

VPC ID can be found in the Outputs of the root stack – eksctl-eks-alb-testing-cluster (check the AWS: CloudFormation – Nested Stacks and stacks parameters Import/Export post about how those stacks were created):

So the file has to look like the next:

Apply it to run ALB Ingress Controller:

[simterm]

$ kubectl apply -f alb-ingress-controller.yaml
deployment.apps/alb-ingress-controller created

[/simterm]

Check its pod:

[simterm]

$ kubectl -n kube-system get pod
NAME                                     READY   STATUS    RESTARTS   AGE
alb-ingress-controller-c47448d84-49tsc   1/1     Running   0          36s

[/simterm]

And the controller’s logs:

[simterm]

$ kubectl logs -n kube-system deployment.apps/alb-ingress-controller -f
-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.1.4
...
I0325 14:18:03.203499       1 leaderelection.go:205] attempting to acquire leader lease  kube-system/ingress-controller-leader-alb...
I0325 14:18:03.211928       1 leaderelection.go:214] successfully acquired lease kube-system/ingress-controller-leader-alb
I0325 14:18:03.312309       1 controller.go:134] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "controller"="alb-ingress-controller"
I0325 14:18:03.412486       1 controller.go:154] kubebuilder/controller "level"=0 "msg"="Starting workers"  "controller"="alb-ingress-controller" "worker count"=1

[/simterm]

Well – it is started, looks like even works – nice!

But is it really working? Let’s test.

ALB Ingress controller Testing

For the test – let’s spin up an additional deployment which will run a pod with NGINX and the Kubernetes Ingress Service in front of it.

So, need to create:

  1. Kubernetes deployment for pods
  2. a NodePort Service to route traffic to pods
  3. and Ingress object with the internet-facing ALB annotation – this has to trigger our ALB Controller to create an AWS Application Load Balancer

Apply:

[simterm]

$ kubectl apply -f nginx-testing-app.yml 
deployment.apps/eks-alb-testing-deployment created
service/eks-alb-testing-web-service created
ingress.extensions/eks-alb-testing-web-ingress created

[/simterm]

Check the controller logs:

[simterm]

E0325 14:24:06.627453       1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="no object matching key \"default/eks-alb-testing-web-ingress\" in local store"  "controller"="alb-ingress-controller" "request"={"Namespace":"default","Name":"eks-alb-testing-web-ingress"}
I0325 14:24:09.257834       1 security_group.go:36] default/eks-alb-testing-web-ingress: creating securityGroup dbd770cc-default-eksalbtes-09fa:managed LoadBalancer securityGroup by ALB Ingress Controller
...
I0325 14:24:09.665207       1 loadbalancer.go:191] default/eks-alb-testing-web-ingress: creating LoadBalancer dbd770cc-default-eksalbtes-09fa
...
I0325 14:24:10.636180       1 targets.go:80] default/eks-alb-testing-web-ingress: Adding targets to arn:aws:elasticloadbalancing:eu-north-1:534***385:targetgroup/dbd770cc-fe1e32fc96596dcf2c1/ef783b9fead58965: i-0622ff000570f32c7:31095, i-062329ba6419ba845:31095
I0325 14:24:10.821148       1 listener.go:110] default/eks-alb-testing-web-ingress: creating listener 80
...
I0325 14:24:13.171590       1 rules.go:82] default/eks-alb-testing-web-ingress: modifying rule 1 on arn:aws:elasticloadbalancing:eu-north-1:534***385:listener/app/dbd770cc-default-eksalbtes-09fa/979c262ea2e12983/455a07658116adcd
I0325 14:24:13.192456       1 rules.go:98] default/eks-alb-testing-web-ingress: rule 1 modified with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]

[/simterm]

Check Ingress resources:

[simterm]

$ kubectl get ingress -o wide
NAME                          HOSTS   ADDRESS                                                                   PORTS   AGE
eks-alb-testing-web-ingress   *       dbd770cc-default-eksalbtes-09fa-1532296804.eu-north-1.elb.amazonaws.com   80      114s

[/simterm]

And try to connect to it:

[simterm]

$ curl -I dbd770cc-default-eksalbtes-09fa-1532296804.eu-north-1.elb.amazonaws.com
HTTP/1.1 200 OK
Date: Wed, 25 Mar 2020 14:26:27 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Server: nginx/1.17.9
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes

[/simterm]

Done.