Redis: Can’t open the log file: Read-only file system

By | 04/03/2019
 

Have a new Redis service which has to run alongside with one already existing

During starting this new service – it fails with the Can’t open the log file: Read-only file system error, although /var/log and log file itself have correct permissions.

The root cause was not very obvious so in this post – its description and solution.

The issue

The full error message is next:

--- Unit redis-cluster.service has begun starting up.
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: *** FATAL CONFIG FILE ERROR ***
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: Reading the configuration file, at line 11
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: >>> ‘logfile /var/log/redis/redis-cluster.log’
Apr 03 10:12:28 bttrm-dev-console redis-server[18010]: Can’t open the log file: Read-only file system
Apr 03 10:12:28 bttrm-dev-console systemd[1]: redis-cluster.service: Control process exited, code=exited status=1
Apr 03 10:12:28 bttrm-dev-console systemd[1]: Failed to start Redis relication cluster node.
--- Subject: Unit redis-cluster.service has failed

The configuration

The log file is set in the /etc/redis-cluster/redis-cluster-master.conf config file:

...
logfile /var/log/redis/redis-cluster.log
...

The new Redis instance is starting using a self-written systemd unit-file /lib/systemd/system/redis-server.service which was copied from alred existing one:

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)

[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf
PIDFile=/var/run/redis/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=redis

ExecStartPre=-/bin/run-parts --verbose /etc/redis/redis-server.pre-up.d
ExecStartPost=-/bin/run-parts --verbose /etc/redis/redis-server.post-up.d
ExecStop=-/bin/run-parts --verbose /etc/redis/redis-server.pre-down.d
ExecStop=/bin/kill -s TERM $MAINPID
ExecStopPost=-/bin/run-parts --verbose /etc/redis/redis-server.post-down.d

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/var/run/redis
CapabilityBoundingSet=~CAP_SYS_PTRACE

# redis-server writes its own config file when in cluster mode so we allow
# writing there (NB. ProtectSystem=true over ProtectSystem=full)
ProtectSystem=true
ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target
Alias=redis.service

The new file is saved as /etc/systemd/system/redis-cluster.service and now looks like this:

[Unit]
Description=Redis relication cluster node
After=network.target

[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis-cluster/redis-cluster-master.conf
PIDFile=/var/run/redis/redis-cluster.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=redis-cluster

ExecStop=/bin/kill -s TERM $MAINPID

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis-cluster
ReadWriteDirectories=-/var/log/redis-cluster
ReadWriteDirectories=-/var/run/redis-cluster
CapabilityBoundingSet=~CAP_SYS_PTRACE

ProtectSystem=true
ReadWriteDirectories=-/etc/redis-cluster

[Install]
WantedBy=multi-user.target

The cause and the solution

Exactly the problem was cause by the ReadWriteDirectories parameter in the unit-file: during coping an exisitng Redi’s unit-file – I just updated path names in the new one from redis to redis-cluster, and among others – the /var/log/redis-cluster path.

So the solution is to change the ReadWriteDirectories=-/var/log/redis-cluster to the  ReadWriteDirectories=-/var/log/redis if you have logfile /var/log/redis/redis-cluster.log in the new Redis config file or vice versa.

I decided to not to break “the beauty” of the path-names in the unit-file and make a new logs directory:

root@bttrm-dev-console:/home/admin# mkdir /var/log/redis-cluster
root@bttrm-dev-console:/home/admin# chown redis:redis /var/log/redis-cluster

Then updated /etc/redis-cluster/redis-cluster-master.conf:

root@bttrm-dev-console:/home/admin# cat /etc/redis-cluster/redis-cluster-master.conf | grep logfile
logfile /var/log/redis-cluster/redis-cluster.log

And check now:

root@bttrm-dev-console:/home/admin# systemctl start redis-cluster.service
root@bttrm-dev-console:/home/admin# systemctl status redis-cluster.service
● redis-cluster.service - Redis relication cluster node
Loaded: loaded (/etc/systemd/system/redis-cluster.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2019-04-03 10:39:42 EEST; 6s ago
Process: 18501 ExecStart=/usr/bin/redis-server /etc/redis-cluster/redis-cluster-master.conf (code=exited, status=0/SUCCESS)
Main PID: 18502 (redis-server)
Tasks: 3 (limit: 4915)
Memory: 1.1M
CPU: 11ms
CGroup: /system.slice/redis-cluster.service
└─18502 /usr/bin/redis-server 0.0.0.0:6389
Apr 03 10:39:42 bttrm-dev-console systemd[1]: Starting Redis relication cluster node...
Apr 03 10:39:42 bttrm-dev-console systemd[1]: redis-cluster.service: PID file /var/run/redis/redis-cluster.pid not readable (yet?) after start: No such file or directory
Apr 03 10:39:42 bttrm-dev-console systemd[1]: Started Redis relication cluster node.

Done.