Redis: running Master-Slave replication in Kubernetes

By | 09/29/2020
 

The task is to spin up a Redis instance in a Kubernetes cluster.

Will use the Master-Slave replication setup with Sentinels for monitoring and failover operations.

Check the Redis: replication, part 2 – Master-Slave replication, and Redis Sentinel post for more details.

Redis cluster vs Redis replication

See Redis: replication, part 1 – an overview. Replication vs Sharding. Sentinel vs Cluster. Redis topology. and Choose between Redis Helm Chart and Redis Cluster Helm Chart.

In short:

  • Replica – includes a Redis Master instance that performs read-write operations and copies data to its Redis Slaves instance(s) which serves Read-only operations. During this, such a Salve can be promoted to the Master’s role if its Master fails.
  • Cluster – have a sense when your Redis have more data than your server’s RAM. The Cluster can use Sharding and a client requesting a piece of data will be redirected to a node that keeps that data.

Ways to run Redis in Kubernetes

Let’s see how we can perform the task – to run a Redis with replication in a Kubernetes cluster.

In our current case, we don’t need to worry about data’s persistence as our Redis will be used as a cache service only so we don’t need for a Kubernetes PersistentVolume.

Helm chart deploy

At first, we will run Redis services from the chart, will take a short look onto them, and then will proceed to the available parameters.

Add the Bitnami repository to your Helm:

helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

Deploy the Redis chart:

helm install backend-redis bitnami/redis
NAME: backend-redis
LAST DEPLOYED: Tue Sep 22 14:48:02 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
Redis can be accessed via port 6379 on the following DNS names from within your cluster:
backend-redis-master.default.svc.cluster.local for read/write operations
backend-redis-slave.default.svc.cluster.local for read-only operations
To get your password run:
export REDIS_PASSWORD=$(kubectl get secret --namespace default backend-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
To connect to your Redis server:
1. Run a Redis pod that you can use as a client:
kubectl run --namespace default backend-redis-client --rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis:6.0.8-debian-10-r0 -- bash
2. Connect using the Redis CLI:
redis-cli -h backend-redis-master -a $REDIS_PASSWORD
redis-cli -h backend-redis-slave -a $REDIS_PASSWORD
To connect to your database from outside the cluster execute the following commands:
kubectl port-forward --namespace default svc/backend-redis-master 6379:6379 &
redis-cli -h 127.0.0.1 -p 6379 -a $REDIS_PASSWORD

Get the password which was generated during the chart’s deployment:

kubectl get secret --namespace default backend-redis -o jsonpath="{.data.redis-password}" | base64 --decode
TySS43UhAW

Run port-forward to connect to the Redis Master instance:

kubectl port-forward --namespace default svc/backend-redis-master 6379:6379
Forwarding from [::1]:6379 -> 6379
Forwarding from 127.0.0.1:6379 -> 6379

Connect:

redis-cli -h 127.0.0.1 -p 6379 -a TySS43UhAW
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379>

“It works!” (с)

Check which Services do we have here:

kk get svc redis-service
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
redis-service   NodePort   172.20.119.242   <none>        6379:32445/TCP   117d

Its type is the NodePort while we need for a  LoadBalancer.

Also, we need for Redis Sentinel, which is off by default:

sentinel.enabled Enable sentinel containers false

Okay, let’s go to the options to enable Sentinel and configure a LoadBalancer.

Redis Options

Create a list of options that can be useful for you.

In my case this will be:

  • global.redis.password
  • metrics:
    • metrics.enabled
    • metrics.serviceMonitor.enabled
    • metrics.serviceMonitor.namespace
  • Persiste:
    • master.persistence.enabled
    • slave.persistence.enabled
  • Service
    • master.service.type
    • master.service.annotations
    • slave.service.type
    • slave.service.annotations
  • Sentinel
    • sentinel.enabled
    • sentinel.service.type – need for LBso clients can ask for Master/Slaves
    • sentinel.service.annotations

Create a new file ~/Temp/redis-opts.yaml to keep our desired parameters:

global:
  redis:
    password: "blablacar"

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    namespace: "monitoring"

master:
  persistence:
    enabled: false
  service:
    type: LoadBalancer
    annotations: 
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internal 

slave:
  persistence:
    enabled: false
  service:
    type: LoadBalancer
    annotations:
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internal

sentinel:
  enabled: true
  service:
    type: LoadBalancer
    annotations: 
      kubernetes.io/ingress.class: alb
      alb.ingress.kubernetes.io/scheme: internal

Update the deployment with -f to specify our parameters file:

helm upgrade --install backend-redis bitnami/redis -f ~/Temp/redis-opts.yaml

Check the password:

kubectl get secret --namespace default backend-redis -o jsonpath="{.data.redis-password}" | base64 --decode
blablacar

And load-balancer:

kk get svc -l app=redis
NAME                     TYPE           CLUSTER-IP       EXTERNAL-IP                                                           PORT(S)                          AGE
backend-redis            LoadBalancer   172.20.204.235   af5e1294a4a73426692c7e25f7bb947d-915967.us-east-2.elb.amazonaws.com   6379:30647/TCP,26379:32523/TCP   80s
backend-redis-headless   ClusterIP      None             <none>                                                                6379/TCP,26379/TCP               80s
backend-redis-metrics    ClusterIP      172.20.66.2      <none>                                                                9121/TCP                         80s

Connect:

redis-cli -h af5e1294a4a73426692c7e25f7bb947d-915967.us-east-2.elb.amazonaws.com -a blablacar
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
af5e1294a4a73426692c7e25f7bb947d-915967.us-east-2.elb.amazonaws.com:6379>

Works, okay.

But why the LoadBalancer is Public when it was set to be internal?

This because the alb.ingress.kubernetes.io/scheme: internal is used for the ALB Ingress Controller while the chart creates a simple Kubernetes Service with the  LoadBalancer type which will create an AWS Classic Load Balancer.

Read the documentation –  https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer and update our annotations: instead of the “alb.ingress.kubernetes.io/scheme: internal” – specify the “service.beta.kubernetes.io/aws-load-balancer-internal: "true"“:

...
master:
  service:
    type: LoadBalancer
    annotations: 
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

slave:
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

sentinel:
  enabled: true
  service:
    type: LoadBalancer
    annotations: 
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Update the deployment and check again:

kk get svc -l app=redis
NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP                                                                        PORT(S)                          AGE
backend-redis            LoadBalancer   172.20.178.72   internal-***-1786016015.us-east-2.elb.amazonaws.com   6379:31192/TCP,26379:32239/TCP   17s
backend-redis-headless   ClusterIP      None            <none>                                                                             6379/TCP,26379/TCP               17s
backend-redis-metrics    ClusterIP      172.20.35.163   <none>                                                                             9121/TCP                         17s

Now, make a Write operation – must be served by the Redis Master instance:

admin@bttrm-dev-app-1:~$ redis-cli -h internal-***-1786016015.us-east-2.elb.amazonaws.com -p 6379 -a blablacar SET testkey testvalue
OK

And a Read operation:

admin@bttrm-dev-app-1:~$ redis-cli -h internal-***-1786016015.us-east-2.elb.amazonaws.com -p 6379 -a blablacar GET testkey
"testvalue"

Replication status:

admin@bttrm-dev-app-1:~$ redis-cli -h internal-***-1786016015.us-east-2.elb.amazonaws.com -p 6379 -a blablacar info replication
Replication
role:master
connected_slaves:1
slave0:ip=10.3.50.119,port=6379,state=online,offset=144165,lag=1
...

And Sentinel status:

admin@bttrm-dev-app-1:~$ redis-cli -h internal-***-1786016015.us-east-2.elb.amazonaws.com -p 26379 -a blablacar info sentinel
Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.3.33.107:6379,slaves=1,sentinels=2

Although write operations need to be performed after getting the Master’s address via Sentinels:

admin@bttrm-dev-app-1:~$ redis-cli -h internal-***-1786016015.us-east-2.elb.amazonaws.com -p 26379 -a blablacar sentinel get-master-addr-by-name mymaster
1) "10.3.33.107"
2) "6379"

Check the documentation.

Done.



Also published on Medium.