Istio: an overview and running Service Mesh in Kubernetes

By | 04/11/2021
0 Comment
 

Istio is a Service Mesh solution that allows performing Service Discovery, Load Balancing, traffic control, canary rollouts and blue-green deployments, traffic monitoring between microservices.

We will use Istio in our AWS Elastic Kubernetes Service for traffic monitoring, as an API Gateway service, for traffic policies, and for various deployment strategies.

In this post, will speak about the Service Mesh concept in general, then will take an overview of Istio architecture and components, its installation process, and how to run a test application.

Service Mesh

Essentially, it’s a proxy-services manager. As a proxy service there can be systems like NGINX, HAProxy, or Envoy, working on the Network OSI Layer 7, that allows for dynamic traffic control and applications communication configuration.

Service mesh performs a new applications/services discovery, load-balancing, authentication, and traffic encryption.

For the traffic control in a service mesh for each application or in the case of Kubernetes for each pod, a proxy service called sidecar is started alongside the application.

Together those sidecar containers are known as a Data Plane.

For their configuration and management, there is another process group called Control Plane. They are used for new applications discovery, encryption keys management, metrics collection and aggregation, and so on.

A service mesh can be displayed as such a scheme:

Among many Service mesh solutions I’d mention the followings:

Read more (all links in Rus):

Istio architecture

So, Istio as a service mesh consists of two main parts – the Data plane and Control plane:

  • Data plane (“a data layer”): contains a collection of proxy services represented as sidecar containers in each Kubernetes Pod, using an extended Envoy proxy server. Those sidecars links and controls traffic between applications, collects and sens metrics
  • Control plane (“control layer”): manages and configures sidecars, aggregates monitoring metrics, TLS certificates management

Isitio architecture can be represented as the following diagram:

Or another one:

Control Plane

Istio Control Plane inculdes for main componentes:

  • Pilot: central controller responsible for communication with sidecars using Envoy API. It reads rules described in istio manifests and sends them to Envoy proxies to configure them. Also, it is used for the service discovery, traffic control, routing, network resistance ability with timeouts and circuit breaking.
  • Citadel: Identity and Access management – traffic encryption, users and services authentification, TLS keys management. See Istio Security.
  • Galley: configuration management – validates new configs and sends them over mesh
  • Mixer: monitoring, metrics, logs, traffic control

Since the Istio version 1.9.1 they all, excluding Pilot, which is running in a dedicated Docker container, are built as a single binary file called istiod, plus additional Ingress and Egress Controllers.

Data Plane

Consist of sidecar containers running as additional processes which are running in Kubernetes via kube-inject, see the Installing the Sidecar.

These containers and Envoy proxy instances allow to:

Read more:

Istio network model

Before going to run our Istio – let’s take a brief overview of the resources used to manage traffic.

During installation, Istio creates an Ingress Gateway service (and Egress Gateway, if this was set during the installation) – a new Kubernetes object described as Kubernetes CRD during Istio install.

In AWS with default settings, Ingress Gateway will create an AWS Classic LoadBalancer, as Ingress Gateway is represented as Kubernetes Ingress object with the LoadBalancer type.

“Behind” an Ingress Gateway is another resource is created – the Gateway, which is also described as Kubernetes CRD during installation and used to describe hosts and ports used to send traffic with this Gateway.

Then, another resource is going to the scene – the VirtualService, which describes traffic routing sent over a Gateway and sends it to Kubernetes Services, as it’s described in the Destination rules.

Ingress Gateway consists of two parts: a Kubernetes Pod with Envoy instance which controls traffic, and a Kubernetes Ingress, which accepts new connections.

In its turn Gateway and VirtualService configures Envoy proxy instance used as the Ingress Gateway controller.

So, in general, traffic flow is the following:

  1. a packet is coming to an external Load Balancer and then sent to a Kubernetes WorkerNode TCP port
  2. there the packet is sent to the Istio IngressGateway Service
  3. and is redirected to the Istio IngressGateway Pod
  4. an Envoy instance on this Pod is configured with Gateway and VirtualService
    1. a Gateway describes ports, protocols, SSL certificates
    2. VirtualService describes traffic routing to a Kubernetes Service of our application
  5. Istio IngressGateway Pod sends this packet to the application’s Service
  6. and the Service routes the packet to an application’s Pod

Read more:

Running Istio in Kubernetes

Istio supports different Deployment models. In this current case, we are using one AWS Elastic Kubernetes Service cluster, and all pods are running in the same VPC network.

Also, there are different Config Profiles which has pre-configured components set to be installed. Here, we can be interested in the default – install istiod and istio-ingressgateway, demo – similar, but also will install theistio-egressgateway service, and preview – to investigate new abilities that are not yet included in the main Istion release.

You can install Istio in various ways: using the istioctl utility and by using manifest files, using Helm, or with Ansible.

Worth paying attention to the Mutual TLS (mTLS) – see the Permissive mode. In short: by default, Istion is installed in the Permissive mTLS mode, which allows existing applications to communicate using plaintext traffic. But all new connections via Envoy proxies will be made with TLS encryption.

For now, let’s install Istio manually with istioctl, and later on the Dev and Production clusters will install with Ansible and its helm module.

Read more:

Download Istio:

[simterm]

$ curl -L https://istio.io/downloadIstio | sh -
$ cd istio-1.9.1/

[/simterm]

istioctl uses the ~/.kube/config file or you can specify your own by using the --kubeconfig option, or set another context with --context.

The istioctl file located in the bin – add it to the $PATH variable (Linux/macOS:

[simterm]

$ export PATH=$PWD/bin:$PATH

[/simterm]

Check if it’s working:

[simterm]

$ istioctl version
no running Istio pods in "istio-system"
1.9.1

[/simterm]

Generate a kubeconfig for the testing cluster, if need (if using Minicube, it will be generated automatically):

[simterm]

$ aws eks update-kubeconfig --name bttrm-eks-test-1-18 --alias iam-bttrm-eks-root-role-kubectl@bttrm-eks-test-1-18
Added new context iam-bttrm-eks-root-role-kubectl@bttrm-eks-test-1-18 to /home/setevoy/.kube/config

[/simterm]

And install Istio with the default profile:

[simterm]

$ istioctl install --set profile=default -y
✔ Istio core installed                                                                                                                                                                                                                       
✔ Istiod installed                                                                                                                                                                                                                           
✔ Ingress gateways installed                                                                                                                                                                                                                 
✔ Installation complete

[/simterm]

Check versions again:

[simterm]

$ istioctl version
client version: 1.9.1
control plane version: 1.9.1
data plane version: 1.9.1 (1 proxies)

[/simterm]

And Istio pods:

[simterm]

$ kubectl -n istio-system get pod
NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-d45fb4b48-jsz9z   1/1     Running   0          64s
istiod-7475457497-6xskm                1/1     Running   0          77s

[/simterm]

Now, let’s deploy a test application and will configure routing via Istio Ingress Gateway.

Running test application

We will not use the default Bookinfo from the Istio Gettings Started guide, instead let’s define our own Namespace, a Deployment with one pod with NGINX, and a Service – I’d like to emulate already existing applications that need to be migrated under Istio control.

Also, at this moment we will not configure automated sidecars injection – will go back to this later.

A manifest looks like the following:

---
apiVersion: v1
kind: Namespace
metadata:
  name: test-namespace
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  namespace: test-namespace
  labels:
    app: test
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
        version: v1
    spec:
      containers:
      - name: web
        image: nginx
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        readinessProbe:
          httpGet:
            path: /
            port: 80
      nodeSelector:
        role: common-workers
---
apiVersion: v1
kind: Service
metadata:
  name: test-svc
  namespace: test-namespace
spec:
  selector:
    app: test
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80

It’s recommended to use the version label in applications, as later this will allow implementing canary and blue-green deployment, see the How To Do Canary Deployments With Istio and Kubernetes и Traffic Management.

Deploy it:

[simterm]

$ kubectl apply -f test-istio.yaml 
namespace/test-namespace created
deployment.apps/test-deployment created
service/test-svc created

[/simterm]

Check containers in the pod that we created:

[simterm]

$ kk -n tkk -n test-namespace get pod -o jsonpath={.items[*].spec.containers[*].name}
web

[/simterm]

Okay, one pod as it was described in the Deployment above.

Using kubectl port-forward connect to the Service:

[simterm]

$ kubectl -n test-namespace port-forward services/test-svc 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

[/simterm]

And check if our application is working:

[simterm]

$ curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

[/simterm]

Nice – here is everything working.

Istio Ingress Gateway

So, at this moment we have an Istio Ingress Gateway created during Istio installation, which is represented by an AWS Classic LoadBalancer:

But if try to access it right now, we will get an error as it has no idea where to route traffic to:

[simterm]

$ curl a6f***037.eu-west-3.elb.amazonaws.com
curl: (52) Empty reply from server

[/simterm]

To configure it – need to add an Istio Gateway.

Gateway

The Gateway describes Istio Ingress Gateway config – which ports to use and which traffic to accept. Also, here you can perform SSL termination (but in this case, this is done by an AWS LoadBalancer).

Add a new resource to our test-istio.yaml – the Gateway:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: test-namespace
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

In the spec.selector.istio we are specifying the Istio Ingress Gateway, for which this manifest will be applied to.

Pay attention, that our application is living in a dedicated namespace, so Gateway and VirtualService (see below) need to be created in the same Namespace.

Create the Gateway:

[simterm]

$ kubectl apply -f test-istio.yaml
namespace/test-namespace unchanged
deployment.apps/test-deployment unchanged
service/test-svc unchanged
gateway.networking.istio.io/test-gateway created

[/simterm]

Check it:

[simterm]

$ kubectl -n test-namespace get gateways
NAME           AGE
test-gateway   17s

[/simterm]

VirtualService

Next, we are going to add a VirtualService where we will describe our “backend” where traffic will be sent.

As a “backend” here is just a common Kubernetes Service of our application – the test-svc:

[simterm]

$ kk -n test-namespace get svc
NAME       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
test-svc   NodePort   172.20.195.107   <none>        80:31581/TCP   15h

[/simterm]

Describe a VirtualService in the same namespace where our application and where we’ve created the Gateway above:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: test-virtualservice
  namespace: test-namespace
spec:
  hosts:
  - "*"
  gateways:
  - test-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: test-svc
        port:
          number: 80

Create it:

[simterm]

$ kubectl apply -f test-istio.yaml
namespace/test-namespace unchanged
deployment.apps/test-deployment unchanged
service/test-svc unchanged
gateway.networking.istio.io/test-gateway unchanged
virtualservice.networking.istio.io/test-virtualservice created

[/simterm]

Check:

[simterm]

$ kubectl -n test-namespace get virtualservice
NAME                  GATEWAYS         HOSTS   AGE
test-virtualservice   [test-gateway]   [*]     42s

[/simterm]

And check again the URL of the LoadBalancer of the Istio Ingress Gateway:

[simterm]

$ curl a6f***037.eu-west-3.elb.amazonaws.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

[/simterm]

And still, we are working without Envoy proxy aka sidecar in our application’s Pod:

[simterm]

$ kk -n tkk -n test-namespace get pod -o jsonpath={.items[*].spec.containers[*].name}
web

[/simterm]

Will check a bit later why this is working.

Kiali – traffic observation

Istio has a lot of addons: Prometheus for metrics collection and alerting, Grafana for metrics visualization, Jaeger for requests tracing, and Kiali to build a map of network and services. Read more at Integrations.

Install all addons:

[simterm]

$ kubectl apply -f samples/addons

[/simterm]

And execute istioctl dashboard kiali – Kiali will be opened in a default browser:

But, if navigate to the Applications, we will see that our application marked as “Missing Sidecar“:

And no services map the Graph:

Sidecar – Envoy proxy

As we remember, our pod has no container with Envoy instance although the network is working, because at this moment Istio via iptables rules sends traffic directly to the container with NGINX. Read more at Traffic flow from application container to sidecar proxy.

Iptalbes rules are configured with an additional InitContainer – istio-init – when a pod is started, but at this moment they are default, configured with kube-proxy during our application deployment. Check the Kubernetes: Service, load balancing, kube-proxy, and iptables post for more details.

In the following post we will dive deeper into the Istio networking, and for now, let’s just add sidecars injection to our Namespace’s Pods and compare Iptables rules before and after.

Istio and iptables

Go to check Iptables rules before we will add a sidecar and istio-init.

Connect via SSH to a Kubernetes WorkerNode where your Pod is living and find NGINX Docker container:

[simterm]

[root@ip-10-22-35-66 ec2-user]# docker ps | grep nginx
22d64b132490        nginx                                                                   "/docker-entrypoint.…"   3 minutes ago       Up 3 minutes                            k8s_web_test-deployment-6864c5bf84-mk98r_test-namespace_8b88caf0-237a-4c94-ac71-186f8e701a7c_0

[/simterm]

Find a PID of the process in this container:

[simterm]

[root@ip-10-22-35-66 ec2-user]# docker top k8s_web_test-deployment-6864c5bf84-mk98r_test-namespace_8b88caf0-237a-4c94-ac71-186f8e701a7c_0
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                31548               31517               0                   10:36               ?                   00:00:00            nginx: master process nginx -g daemon off;
101                 31591               31548               0                   10:36               ?                   00:00:00            nginx: worker process

[/simterm]

Using the nsenter utility check Ipables rules on the network-namespace of the process with PID 31548 – nothing unusual here, for now – all traffic is just sent directly to our container:

[simterm]

[root@ip-10-22-35-66 ec2-user]# nsenter -t 31548 -n iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

[/simterm]

Running Sidecar

Documentation – Installing the Sidecar.

To automatically inject Envoy proxy instances to pods in the test-namespace namespace run the following:

[simterm]

$ kubectl label namespace test-namespace istio-injection=enabled
namespace/test-namespace labeled

[/simterm]

Check the labels:

[simterm]

$ kubectl get namespace test-namespace --show-labels
NAME             STATUS   AGE   LABELS
test-namespace   Active   11m   istio-injection=enabled

[/simterm]

But sidecars will be added only for new pods in this namespace.

We can add them manually by using the kube-inject, or just by recreating the pod:

[simterm]

$ kubectl  -n test-namespace scale deployment test-deployment --replicas=0
deployment.apps/test-deployment scaled

$ kubectl  -n test-namespace scale deployment test-deployment --replicas=1
deployment.apps/test-deployment scaled

[/simterm]

Check containers in the Pod again:

[simterm]

$ kubectl -n test-namespace get pod -o jsonpath={.items[*].spec.containers[*].name}
web istio-proxy

[/simterm]

Now we can see the istio-proxy container – this is pour sidecar container with Envoy.

Also, check initContainers of the Pod:

[simterm]

$ kk -n test-namespace get pod -o jsonpath={.items[*].spec.initContainers[*].name}
istio-init

[/simterm]

And check Iptables rules again – find the container, its PID, and check the rules:

[simterm]

[root@ip-10-22-35-66 ec2-user]# nsenter -t 4194 -n iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
ISTIO_INBOUND  tcp  --  anywhere             anywhere            

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ISTIO_OUTPUT  tcp  --  anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain ISTIO_INBOUND (1 references)
target     prot opt source               destination         
RETURN     tcp  --  anywhere             anywhere             tcp dpt:15008
RETURN     tcp  --  anywhere             anywhere             tcp dpt:ssh
RETURN     tcp  --  anywhere             anywhere             tcp dpt:15090
RETURN     tcp  --  anywhere             anywhere             tcp dpt:15021
RETURN     tcp  --  anywhere             anywhere             tcp dpt:15020
ISTIO_IN_REDIRECT  tcp  --  anywhere             anywhere            

Chain ISTIO_IN_REDIRECT (3 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             redir ports 15006

Chain ISTIO_OUTPUT (1 references)
target     prot opt source               destination         
RETURN     all  --  ip-127-0-0-6.eu-west-3.compute.internal  anywhere            
ISTIO_IN_REDIRECT  all  --  anywhere            !localhost            owner UID match 1337
RETURN     all  --  anywhere             anywhere             ! owner UID match 1337
RETURN     all  --  anywhere             anywhere             owner UID match 1337
ISTIO_IN_REDIRECT  all  --  anywhere            !localhost            owner GID match 1337
RETURN     all  --  anywhere             anywhere             ! owner GID match 1337
RETURN     all  --  anywhere             anywhere             owner GID match 1337
RETURN     all  --  anywhere             localhost           
ISTIO_REDIRECT  all  --  anywhere             anywhere            

Chain ISTIO_REDIRECT (1 references)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             redir ports 15001

[/simterm]

Now we can see additional Iptables chains and rules created by Isio, which sends traffic to the Envoy sidecar container and then to the application’s container and back.

Go back to the Kiali dashboard and check the map:

And requests tracing:

Traces are also available via the Jaeger dashboard which can be accessed with istioctl dashboard jaeger:

Also, with istioctl dashboard prometheus you can open Prometheus and check available metrics, see more at Querying Metrics from Prometheus:

Actually, that’s all for now.

Later, we will integrate Istio with AWS Application LoadBalncer, deploy and configuration of an Isitio instance to the AWS Elastic Kubernetes Service with Ansible and Helm, Gateway and VirtualServices configuration and debugging.

Useful links