The whole story was started when we decided to get rid of memcached.
Currently, we have memcahced and Redis running on our backend servers.
And memcached, and Redis instances are working as standalone applications, i.e. they are not connected in any kind of replication and this leads to a problem:
we have three backend hosts which are hosted behind an AWS Application Load Balancer
an ALB has Sticky Sessions enabled but it’s working with cookies which are ignored by our mobile applications (iOS/Android)
respectively, when a client makes a request to the backend – sometimes it can get cached data which already was removed/updated on another backend host in Redis ormemcached
We have this scheme since we migrated our backend application from an old infrastructure where the only one host was used and still had no time to update it, although it was in our planes a long time.
Currently, to solve these issues we have a bunch of “hacks” on the backend which makes additional checks to ensure data is up to date, and now to get rid of them we decided to:
get rid of memcached at all as Redis can be used for the functions where memcached is used now
configure Redis replication over all hosts
Such a setup will be described in the post below.
The first example – with a basic Master-Slave replication and the second example – the Sentinel set up and configuration.
AWS EC2 instances with Debian 9 will be used here.
To work with Redis hosts three domain names will be used – redis-0.setevoy.org.ua for a master, redis-1.setevoy.org.ua and redis-2.setevoy.org.ua for its two slaves.
Slave in a minimal setup can be only one but as the second example here will be with the Sentinel – let’s have three from the beginning.
The basic Master-Slave replication
In this way, slaves will be a master’s read-only replicas keeping the same data which will be added to the master.
Master will send all data updates to its slaves – new keys expire, etc.
In case of the link between master and slave will be broken – a slave will try to reconnect to the master and make a partial synchronization to update data from a place where the previous sync was interrupted.
In case of such a partial sync is not possible – the slave will ask the master for a full synchronization and master will perform its data full snapshot which will be sent to this slave after this a usual sync will be restored.
A couple of notes here to keep in mind:
one master can have multitype slaves
slaves can accept connections from other slaves, making kind of “cascade” of a replicated nodes – a master on the top, a slave(s) in the middle and a slave(s) at the bottom