Kubernetes: part 2 – a cluster set up on AWS with AWS cloud-provider and AWS LoadBalancer

In the first part – Kubernetes: part 1 – architecture and main components overview – we did a quick glance about Kubernetes.

Also, check the third part – Kubernetes: part 3 – AWS EKS overview and manual EKS cluster set up.

The next thing I’d like to play with is to manually create a cluster using kubeadm, run a simple web-service there and access it via AWS LoadBalancer.

The main issue I faced with during this set up was lack of full-fledged documentation and up to date examples, thus had to do almost everything by the cut-and-try method.

Just to finally see a message saying:

WARNING: aws built-in cloud provider is now deprecated. The AWS provider is deprecated and will be removed in a future release

The example below uses the Kubernetes version: v1.15.2. and ЕС2 with OS Ubuntu 18.04

Preparing AWS

VPC

Create a VPC with the 10.0.0.0/16 CIDR:

Add a tag named kubernetes.io/cluster/kubernetes with the owned value – it will be used by K8s for AWS resources auto-discovery related to the Kubernetes stack, also it will add such a tag itself during creating new resources:

Enable DNS hostnames:

Subnet

Create a new subnet in this VPC:

Enable Public IPs for EC2 instances which will be placed in this subnet:

Add the tag:

Internet Gateway

Create an IGW to route traffic from the subnet into the Internet:

For IGW add the tag as well, just in case:

Attached this IGW to your VPC:

Route Table

Create a routing table:

Add the tag here:

Click on the Routes tab, add a new route to the 0.0.0.0/0 network via the IGW we created above:

Attach this table to the subnet – Edit subnet association:

Choose your subnet created earlier:

IAM role

To make Kubernetes working with AWS need to create two IAM EC2 roles – for master and slaves.

You can also use ACCESS/SECRET instead.

IAM Master role

Go to the IAM > Policies, click Create policy, into the JSON add a new policy description (see cloud-provider-aws):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "ec2:DescribeInstances",
                "ec2:DescribeRegions",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:ModifyInstanceAttribute",
                "ec2:ModifyVolume",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateRoute",
                "ec2:DeleteRoute",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteVolume",
                "ec2:DetachVolume",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DescribeVpcs",
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:AttachLoadBalancerToSubnets",
                "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateLoadBalancerPolicy",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:DeleteLoadBalancerListeners",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DetachLoadBalancerFromSubnets",
                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeLoadBalancerPolicies",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
                "iam:CreateServiceLinkedRole",
                "kms:DescribeKey"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Save it:

Go to the Roles, create a role using the EC2 type:

Click on the Permissions, find and attach the policy added above:

IAM Worker role

In the same way, create another policy for worker nodes:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeRegions",
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
            ],
            "Resource": "*"
        }
    ]
}

Save it as k8s-cluster-iam-worker-policy (can be used any name obviously):

And create a k8s-cluster-iam-master-role:

Running EC2

Create an EC2 using t2.medium type (minimal type as cKubernetes master needs to have at least 2 CPU cores), using your VPC and set k8s-cluster-iam-master-role as the IAM role:

Add tags:

Create a Security Group:

Wile Master is spinning up – creat a Worker Node in the same way just using the k8s-cluster-iam-worker-role:

Tags:

Attach existing SG:

Connect to any instances and check if the network is working:

Good.

A Kubernetes cluster set up

Kubernetes installation

Perform the next steps on both EC2.

Update packages list and installed packages:

Add Docker and Kubernetes repositories:

Or do everything just be one command:

Hostname

Perform the next steps on both EC2.

Afaik the next changes need to be done on Ubuntu only, and you can’t change a hostname which was set by AWS (the ip-10-0-0-102 in this example).

Check a hostname now:

Get it as a fully qualified domain name (FQDN):

Set the hostname as FQDN:

Check now:

Repeat on the worker node.

Cluster set up

Create file /etc/kubernetes/aws.yml:

---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  serviceSubnet: "10.100.0.0/16"
  podSubnet: "10.244.0.0/16"
apiServer:
  extraArgs:
    cloud-provider: "aws"
controllerManager:
  extraArgs:
    cloud-provider: "aws"

Initialize cluster using this config:

Create a kubelet config file:

Check nodes:

You can get your cluster-info using the config view:

kubeadm reset

In case you want to fully destroy your cluster to run set up it from the scratch – use reset:

And reset IPTABLES rules:

Flannel CNI installation

From the Master node execute:

Wait a minute and check nodes again:

STATUS == Ready, Okay.

Attaching the Worker Node

On the Worker node create a /etc/kubernetes/node.yml file with the JoinConfiguration:

---
apiVersion: kubeadm.k8s.io/v1beta1
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: "rat2th.qzmvv988e3pz9ywa"
    apiServerEndpoint: "10.0.0.102:6443"
    caCertHashes:
      - "sha256:ce983b5fbf4f067176c4641a48dc6f7203d8bef972cb9d2d9bd34831a864d744"
nodeRegistration:
  name: ip-10-0-0-186.eu-west-3.compute.internal
  kubeletExtraArgs:
    cloud-provider: aws

Join this node to the cluster:

Go back to the Master, check nodes one more time:

Load Balancer creation

And the last thing is to run a web-service, let’s use a simple NGINX container and to place a LoadBalancer Service:

kind: Service
apiVersion: v1
metadata:
  name: hello
spec:
  type: LoadBalancer
  selector:
    app: hello
  ports:
    - name: http
      protocol: TCP
      # ELB's port
      port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: hello
          image: nginx

Apply it:

Check Deployment:

ReplicaSet:

Pod:

And Services:

Check an ELB in the AWS Console:

Instances – here is our Worker node:

Let’s recall how it’s working:

1. AWS ELB will route traffic to the Worker Node (NodePort Service)
2. on the Worker node via a NodePort service it will be routed to the Pod’s port (TargetPort)
3. on the Pod with the TargetPort traffic will be routed to a container’s port (containerPort)

In the LoadBalancer description above we see the next setting:

Port Configuration
80 (TCP) forwarding to 30381 (TCP)

Check the Kubernetes cluster services:

Наш NodePort: http 30381/TCP

You can send a request directly to the Node.

Find a Worker node’s address:

And connect to the 30381 port:

Check if ELB is working:

A pod’s logs:

AWS Load Balancer – no Worker Node added

While I tried to setup this cluster and ELB faced with the issue when Worker Nodes wasn’t added to an AWS LoadBalancer when creating a LoadBalancer Kubernetes service.

In such a case try to check if the ProviderID (--provider-id) is present in a node’s settings:

If there is no ProviderID add it using the kubectl edit node <NODE_NAME> as a ProviderID: aws:///eu-west-3a/<EC2_INSTANCE_ID>:

But it must be set when you are joining a node using /etc/kubernetes/node.yml file with the JoinConfiguration with the cloud-provider: aws is set.

Done.

Useful links

Common

• Kubernetes by kubeadm config yamls

AWS

• Kubernetes Cloud Controller Manager
• Cloud Providers
• How to create a k8s cluster with kubeadm on Azure
• Setting up the Kubernetes AWS Cloud Provider
• Using Kubeadm to Add New Control Plane Nodes with AWS Integration
• Kubernetes Cluster Setup with AWS Cloud Provider
• Kubernetes on AWS
• Rancher: AWS Cloud Provider

LoadBalancer, network

• Services of type LoadBalancer and Multiple Ingress Controllers
• Kubernetes and AWS ELB — What to do when you reach the Security Group limit in AWS
• Using Kubernetes LoadBalancer Services on AWS
• Deploy an app behind a Load Balancer on Kubernetes
• Load Balancing and Reverse Proxying for Kubernetes Services
• How to Add Load Balancers to Kubernetes Clusters
• AWS ALB Ingress Controller for Kubernetes

Also published on Medium.