Linux: persistent logs in systemd-journald

By | 09/25/2019
 

We have an AWS EC2 which suddenly rebooted.

The issue is that its /var/log/messages incomplete and the latest record is about rsyslog service killed – but nothing about who killed it and why.

Then I went to the journald but surprise – it stores logs only for the current boot so there was no way to find the reboot’s root cause.:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
0 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:48:44 EEST

So, need to configure journald to store logs after each reboot.

Its config file /etc/systemd/journald.conf.

To make logs to be stored always – update the Storage parameter to the auto or persistent.

If auto – then the /var/run/journal directory must be created manually, with the persistentsystemd will create it.

Update config, set persistent, restart systemd-journald to apply:

root@bttrm-stage-console:/home/admin# systemctl restart systemd-journald

Check the logs directory:

root@bttrm-stage-console:/home/admin# ll /var/log/journal/
total 4
drwxr-xr-x 2 root root 4096 Sep 24 17:54 7066493f616c4a6285c19cfaa2884b09
root@bttrm-stage-console:/home/admin# ll /var/log/journal/7066493f616c4a6285c19cfaa2884b09/
total 57348
-rw-r----- 1 root root 58720256 Sep 24 17:54 system.journal

Reboot the EC2 and check logs again:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
-1 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:56:19 EEST
0 591136b94276490580f4e39e576e639e Tue 2019-09-24 17:56:20 EEST—Tue 2019-09-24 17:56:41 EEST

Now you are able to check the previous reboot log:

root@bttrm-stage-console:/home/admin# journalctl -b -1
-- Logs begin at Tue 2019-09-24 12:13:02 EEST, end at Tue 2019-09-24 17:57:36 EEST. --
Sep 24 12:13:02 bttrm-stage-console CRON[19407]: pam_unix(cron:session): session closed for user me-v3

Size can be limited by setting the SystemMaxUse, see the documentation.

Done.