Have a new Redis service which has to run alongside with one already existing
During starting this new service – it fails with the Can’t open the log file: Read-only file system error, although /var/log and log file itself have correct permissions.
The root cause was not very obvious so in this post – its description and solution.
Contents
The issue
The full error message is next:
[simterm]
— Unit redis-cluster.service has begun starting up.
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: *** FATAL CONFIG FILE ERROR ***
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: Reading the configuration file, at line 11
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: >>> ‘logfile /var/log/redis/redis-cluster.log’
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: Can’t open the log file: Read-only file system
Apr 03 10:12:28 bttrm-dev-console systemd[1]: redis-cluster.service: Control process exited, code=exited status=1
Apr 03 10:12:28 bttrm-dev-console systemd[1]: Failed to start Redis relication cluster node.
— Subject: Unit redis-cluster.service has failed
[/simterm]
The configuration
The log file is set in the /etc/redis-cluster/redis-cluster-master.conf config file:
... logfile /var/log/redis/redis-cluster.log ...
The new Redis instance is starting using a self-written systemd unit-file /lib/systemd/system/redis-server.service which was copied from alred existing one:
[Unit] Description=Advanced key-value store After=network.target Documentation=http://redis.io/documentation, man:redis-server(1) [Service] Type=forking ExecStart=/usr/bin/redis-server /etc/redis/redis.conf PIDFile=/var/run/redis/redis-server.pid TimeoutStopSec=0 Restart=always User=redis Group=redis RuntimeDirectory=redis ExecStartPre=-/bin/run-parts --verbose /etc/redis/redis-server.pre-up.d ExecStartPost=-/bin/run-parts --verbose /etc/redis/redis-server.post-up.d ExecStop=-/bin/run-parts --verbose /etc/redis/redis-server.pre-down.d ExecStop=/bin/kill -s TERM $MAINPID ExecStopPost=-/bin/run-parts --verbose /etc/redis/redis-server.post-down.d UMask=007 PrivateTmp=yes LimitNOFILE=65535 PrivateDevices=yes ProtectHome=yes ReadOnlyDirectories=/ ReadWriteDirectories=-/var/lib/redis ReadWriteDirectories=-/var/log/redis ReadWriteDirectories=-/var/run/redis CapabilityBoundingSet=~CAP_SYS_PTRACE # redis-server writes its own config file when in cluster mode so we allow # writing there (NB. ProtectSystem=true over ProtectSystem=full) ProtectSystem=true ReadWriteDirectories=-/etc/redis [Install] WantedBy=multi-user.target Alias=redis.service
The new file is saved as /etc/systemd/system/redis-cluster.service and now looks like this:
[Unit] Description=Redis relication cluster node After=network.target [Service] Type=forking ExecStart=/usr/bin/redis-server /etc/redis-cluster/redis-cluster-master.conf PIDFile=/var/run/redis/redis-cluster.pid TimeoutStopSec=0 Restart=always User=redis Group=redis RuntimeDirectory=redis-cluster ExecStop=/bin/kill -s TERM $MAINPID UMask=007 PrivateTmp=yes LimitNOFILE=65535 PrivateDevices=yes ProtectHome=yes ReadOnlyDirectories=/ ReadWriteDirectories=-/var/lib/redis-cluster ReadWriteDirectories=-/var/log/redis-cluster ReadWriteDirectories=-/var/run/redis-cluster CapabilityBoundingSet=~CAP_SYS_PTRACE ProtectSystem=true ReadWriteDirectories=-/etc/redis-cluster [Install] WantedBy=multi-user.target
The cause and the solution
Exactly the problem was cause by the ReadWriteDirectories parameter in the unit-file: during coping an exisitng Redi’s unit-file – I just updated path names in the new one from redis to redis-cluster, and among others – the /var/log/redis-cluster path.
So the solution is to change the ReadWriteDirectories=-/var/log/redis-cluster to the ReadWriteDirectories=-/var/log/redis if you have logfile /var/log/redis/redis-cluster.log in the new Redis config file or vice versa.
I decided to not to break “the beauty” of the path-names in the unit-file and make a new logs directory:
[simterm]
root@bttrm-dev-console:/home/admin# mkdir /var/log/redis-cluster root@bttrm-dev-console:/home/admin# chown redis:redis /var/log/redis-cluster
[/simterm]
Then updated /etc/redis-cluster/redis-cluster-master.conf:
[simterm]
root@bttrm-dev-console:/home/admin# cat /etc/redis-cluster/redis-cluster-master.conf | grep logfile logfile /var/log/redis-cluster/redis-cluster.log
[/simterm]
And check now:
[simterm]
root@bttrm-dev-console:/home/admin# systemctl start redis-cluster.service
root@bttrm-dev-console:/home/admin# systemctl status redis-cluster.service
● redis-cluster.service - Redis relication cluster node
Loaded: loaded (/etc/systemd/system/redis-cluster.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2019-04-03 10:39:42 EEST; 6s ago
Process: 18501 ExecStart=/usr/bin/redis-server /etc/redis-cluster/redis-cluster-master.conf (code=exited, status=0/SUCCESS)
Main PID: 18502 (redis-server)
Tasks: 3 (limit: 4915)
Memory: 1.1M
CPU: 11ms
CGroup: /system.slice/redis-cluster.service
└─18502 /usr/bin/redis-server 0.0.0.0:6389
Apr 03 10:39:42 bttrm-dev-console systemd[1]: Starting Redis relication cluster node...
Apr 03 10:39:42 bttrm-dev-console systemd[1]: redis-cluster.service: PID file /var/run/redis/redis-cluster.pid not readable (yet?) after start: No such file or directory
Apr 03 10:39:42 bttrm-dev-console systemd[1]: Started Redis relication cluster node.
[/simterm]
Done.
