Redis: replication, part 2 – Master-Slave replication, and Redis Sentinel

By | 03/29/2019

The first part – Redis: replication, part 1 – overview. Replication vs Sharding. Sentinel vs Cluster. Redis topology.

The next part – Redis: replication, part 3 redis-py and work with Redis Sentinel from Python.

The whole story was started when we decided to get rid of memcached.

Currently, we have memcahced and Redis running on our backend servers.

And memcached, and Redis instances are working as standalone applications, i.e. they are not connected in any kind of replication and this leads to a problem:

  • we have three backend hosts which are hosted behind an AWS Application Load Balancer
  • an ALB has Sticky Sessions enabled but it’s working with cookies which are ignored by our mobile applications (iOS/Android)
  • respectively, when a client makes a request to the backend – sometimes it can get cached data which already was removed/updated on another backend host in Redis ormemcached

We have this scheme since we migrated our backend application from an old infrastructure where the only one host was used and still had no time to update it, although it was in our planes a long time.

Currently, to solve these issues we have a bunch of “hacks” on the backend which makes additional checks to ensure data is up to date, and now to get rid of them we decided to:

  1. get rid of memcached at all as Redis can be used for the functions where memcached is used now
  2. configure Redis replication over all hosts

Such a setup will be described in the post below.

The first example – with a basic Master-Slave replication and the second example – the Sentinel set up and configuration.

AWS EC2 instances with Debian 9 will be used here.

To work with Redis hosts three domain names will be used – for a master, and for its two slaves.

Slave in a minimal setup can be only one but as the second example here will be with the Sentinel – let’s have three from the beginning.

The basic Master-Slave replication

In this way, slaves will be a master’s read-only replicas keeping the same data which will be added to the master.

Master will send all data updates to its slaves – new keys expire, etc.

In case of the link between master and slave will be broken – a slave will try to reconnect to the master and make a partial synchronization to update data from a place where the previous sync was interrupted.

In case of such a partial sync is not possible – the slave will ask the master for a full synchronization and master will perform its data full snapshot which will be sent to this slave after this a usual sync will be restored.

A couple of notes here to keep in mind:

  • one master can have multitype slaves
  • slaves can accept connections from other slaves, making kind of “cascade” of a replicated nodes – a master on the top, a slave(s) in the middle and a slave(s) at the bottom
  • it’s strongly recommended to enable data persistence on the master to avoid data loss – see the Safety of replication when master has persistence turned off
  • slave will work in the read-only mode by default, see the Read-only slave

Redis Master configuration

Install Redis:

[email protected]:/home/admin# apt -y install redis-server

Edit a /etc/redis/redis.conf and in the  bind set interfaces to listen on:


You can specify multitype IPs here separated by spaces:


Other valuable options here:

  • port 6379 – clear enough but keep it in mind
  • slave-read-only yes – slaves will be working in the read-only mode, doesn’t affect a master node
  • requirepass foobared – password for master authorization
  • appendonly yes and appendfilename "appendonly.aof" – decrease data loss chance, see the Redis Persistence

Restart the service:

[email protected]:/home/admin# systemctl restart redis

Check it using -a for the password:

[email protected]:/home/admin# redis-cli -a foobared ping

Check data replication status:

[email protected]:/home/admin# redis-cli -a foobared info replication

Add a new data:

[email protected]:/home/admin# redis-cli -a foobared set test 'test'

Get it back:

[email protected]:/home/admin# redis-cli -a foobared get test

Okay – everything works here

Redis Slave configuration

On the two hosts left make a slaves configuration.

It will be the same for both – just repeat it.

Install Redis:

[email protected]:/home/admin# apt -y install redis-server

Edit the /etc/redis/redis.conf:

slaveof 6379
masterauth foobared
requirepass foobared


  • slaveof – set the master’s host and port
  • masterauth – master’s auth
  • requirepass – auth on this replica

Restart the service:

[email protected]:/home/admin# systemctl restart redis

Check its status:

[email protected]:/home/admin# redis-cli -a foobared info replication

Check log:

[email protected]:/home/admin# tail -f /var/log/redis/redis-server.log
16961:S 29 Mar 10:54:36.263 * Connecting to MASTER
16961:S 29 Mar 10:54:36.308 * MASTER <-> SLAVE sync started
16961:S 29 Mar 10:54:36.309 * Non blocking connect for SYNC fired the event.
16961:S 29 Mar 10:54:36.309 * Master replied to PING, replication can continue...
16961:S 29 Mar 10:54:36.310 * Partial resynchronization not possible (no cached master)
16961:S 29 Mar 10:54:36.311 * Full resync from master: 93585eeb7e32c0550c35f8d4935c9a18c4177ab9:1
16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: receiving 92 bytes from master
16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: Flushing old data
16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: Loading DB in memory
16961:S 29 Mar 10:54:36.383 * MASTER <-> SLAVE sync: Finished with success

Connection to the master established, syncronization is done – okay, check the data:

[email protected]:/home/admin# redis-cli -a foobared get test

Data present – all works here as well.

Changing Slave => Master roles

In case of the master will go down – you have to switch one of the slaves to become a new master.

If you’ll try to add any data on a current slave – Redis will rise an error as the slaves are in the read-only mode:

slave-read-only yes

Try to add something:

[email protected]:/home/admin# redis-cli -a foobared set test2 'test2'
(error) READONLY You can't write against a read only slave.

Now connect to the slave:

[email protected]:/home/admin# redis-cli

Authorize:> auth foobared

Disable the slave-role:> slaveof no one

Check its status now:> info replication

Add a new key one more time:> set test2 'test2'

And get it back:> get test2

Keep in mind that as we did those changes in Redis node directly – after its restart it will become a slave again as it’s still is set in its /etc/redis/redis.conf file with the slaveof parameter.

Redis Sentinel

Now let’s add the Sentinel to our replication which will monitor Redis nodes and will do roles switches automatically.

The overall scheme will be next:


  • M1 = Master
  • R1 = Replica 1 / Slave 1
  • R2 = Replica 2 / Slave 2
  • S1 = Sentinel 1
  • S2 = Sentinel 2
  • S3 = Sentinel 3

M1 and S1 – will be on the redis-0, R1 and S2 – on the redis-1, R2 and S3 – on the redis-2.

Running Sentinel

To run a Sentinel daemon the redis-server can be used just with a separate config – /etc/redis/sentinel.conf.

First, let’s create such config file on the Redis Master host:

sentinel monitor redis-test 6379 2
sentinel down-after-milliseconds redis-test 6001
sentinel failover-timeout redis-test 60000
sentinel parallel-syncs redis-test 1
sentinel auth-pass redis-test foobared


  • monitor – the master-node address to be monitored, and the 2 is the Sentinel’s instances number to make a decisions
  • down-after-milliseconds – time after which master will be considered as out of order
  • failover-timeout – time to wait after changing slave=>master roles
  • parallel-syncs – number of simultaneous slaves synchronization after the master changed

Run it:

[email protected]:/home/admin# redis-server /etc/redis/sentinel.conf --sentinel
10447:X 29 Mar 14:15:53.192 * Increased maximum number of open files to 10032 (it was originally set to 1024).
_.-``__ ''-._
_.-``    `.  `_.  ''-._           Redis 3.2.6 (00000000/0) 64 bit
.-`` .-```.  ```\/    _.,_ ''-._
(    '      ,       .-`  | `,    )     Running in sentinel mode
|`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
|    `-._   `._    /     _.-'    |     PID: 10447
`-._    `-._  `-./  _.-'    _.-'
|`-._`-._    `-.__.-'    _.-'_.-'|
|    `-._`-._        _.-'_.-'    | 
`-._    `-._`-.__.-'_.-'    _.-'
|`-._`-._    `-.__.-'    _.-'_.-'|
|    `-._`-._        _.-'_.-'    |
`-._    `-._`-.__.-'_.-'    _.-'
`-._    `-.__.-'    _.-'
`-._        _.-'
10447:X 29 Mar 14:15:53.193 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
10447:X 29 Mar 14:15:53.195 # Sentinel ID is e9fb72c8edb8ec2028e6ce820b9e72e56e07cf1e
10447:X 29 Mar 14:15:53.195 # +monitor master redis-test 6379 quorum 2
10447:X 29 Mar 14:15:53.196 * +slave slave 6379 @ redis-test 6379
10447:X 29 Mar 14:16:43.402 * +slave slave 6379 @ redis-test 6379

Check Sentinel’s status using the 26379 port:

[email protected]:/home/admin# redis-cli -p 26379 info sentinel


  • master0:name=redis-test,status=ok – master is UP
  • slaves=2 – it has two slaves
  • sentinels=1 – only one Sentinel instance is running for now

You can get some basic information here, for example – the master’s IP:

[email protected]:/home/admin# redis-cli -p 26379 sentinel get-master-addr-by-name redis-test
1) ""
2) "6379"

Now repeat Sentinel start on both slaves nodes using the same config as we did on the master and in the Sentinel’s log you must see new instances connected:

10447:X 29 Mar 14:18:40.437 * +sentinel sentinel fdc750c7d6388a6142d9e27b68172f5846e75d8c 26379 @ redis-test 6379
10447:X 29 Mar 14:18:42.725 * +sentinel sentinel ecddb26cd27c9a17c4251078c977761faa7a3250 26379 @ redis-test 6379

Check status again:

[email protected]:/home/admin# redis-cli -p 26379 info sentinel

sentinels=3 – okay.

Also, Sentinel will perform its own settings updates when needed:

[email protected]:/home/admin# cat /etc/redis/sentinel.conf
sentinel myid fdc750c7d6388a6142d9e27b68172f5846e75d8c
sentinel monitor redis-test 6379 2
sentinel down-after-milliseconds redis-test 6001
sentinel failover-timeout redis-test 60000
port 26379
dir "/home/admin"
sentinel auth-pass redis-test foobared
sentinel config-epoch redis-test 0
sentinel leader-epoch redis-test 0
sentinel known-slave redis-test 6379
sentinel known-slave redis-test 6379
sentinel known-sentinel redis-test 26379 ecddb26cd27c9a17c4251078c977761faa7a3250
sentinel known-sentinel redis-test 26379 e9fb72c8edb8ec2028e6ce820b9e72e56e07cf1e
sentinel current-epoch 0

Here is the sentinel myid fdc750c7d6388a6142d9e27b68172f5846e75d8c line added and the whole block after the #Generated by CONFIG REWRITE.

Redis Sentinel Automatic Failover

Now let’s check what will happen if the master will go down.

You can do it manually just by calling kill -9 or by using the redis-cli and sending the DEBUG command with a time in seconds to make a master “down” or by sending a signal to kill the master with.

[email protected]:/home/admin# redis-cli -a foobared DEBUG sleep 30

The Sentinel’s log on the master:

10447:X 29 Mar 14:24:56.549 # +sdown master redis-test 6379
10447:X 29 Mar 14:24:56.614 # +new-epoch 1
10447:X 29 Mar 14:24:56.615 # +vote-for-leader ecddb26cd27c9a17c4251078c977761faa7a3250 1
10447:X 29 Mar 14:24:56.649 # +odown master redis-test 6379 #quorum 3/2
10447:X 29 Mar 14:24:56.649 # Next failover delay: I will not start a failover before Fri Mar 29 14:26:57 2019
10447:X 29 Mar 14:24:57.686 # +config-update-from sentinel ecddb26cd27c9a17c4251078c977761faa7a3250 26379 @ redis-test 6379
10447:X 29 Mar 14:24:57.686 # +switch-master redis-test 6379 6379
10447:X 29 Mar 14:24:57.686 * +slave slave 6379 @ redis-test 6379
10447:X 29 Mar 14:24:57.686 * +slave slave 6379 @ redis-test 6379
10447:X 29 Mar 14:25:03.724 # +sdown slave 6379 @ redis-test 6379

Currently, we are interested in those two lines here:

10384:X 29 Mar 14:24:57.686 # +config-update-from sentinel ecddb26cd27c9a17c4251078c977761faa7a3250 26379 @ redis-test 6379
10384:X 29 Mar 14:24:57.686 # +switch-master redis-test 6379 6379

Sentinel performed the slave-to-master reconfiguration. – is the old master which is dead now, and is a new master, elected from the slaves – it’s running on the redis-1 host.

Try adding data here:

[email protected]:/home/admin# redis-cli -a foobared set test3 'test3'

While a similar attempt on the old master which became a slave now will lead to an error:

[email protected]:/home/admin# redis-cli -a foobared set test4 'test4'
(error) READONLY You can't write against a read only slave.

And let’s kill a node at all and see what Sentinel will do now:

[email protected]:/home/admin# redis-cli -a foobared DEBUG SEGFAULT
Error: Server closed the connection


10447:X 29 Mar 14:26:21.897 * +reboot slave 6379 @ redis-test 6379

Well – Sentinel just restarted that node

Sentinel commands

Command Description
sentinel masters list all masters and their statuses
sentinel master one master’s status
sentinel slaves list all slaves and their statuses
sentinel sentinels list all Sentinel instances and their statuses
sentinel failover run failover manually
sentinel flushconfig force Sentinel to rewrite its configuration on disk
sentinel monitor add a new master
sentinel remove remove master from being watched

Related links

Also published on Medium.