We have an AWS EC2 which suddenly rebooted.
The issue is that its /var/log/messages
incomplete and the latest record is about rsyslog
service killed – but nothing about who killed it and why.
Then I went to the journald
but surprise – it stores logs only for the current boot so there was no way to find the reboot’s root cause.:
[simterm]
root@bttrm-stage-console:/home/admin# journalctl --list-boots 0 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:48:44 EEST
[/simterm]
So, need to configure journald
to store logs after each reboot.
Its config file /etc/systemd/journald.conf
.
To make logs to be stored always – update the Storage
parameter to the auto or persistent.
If auto – then the /var/run/journal
directory must be created manually, with the persistent – systemd
will create it.
Update config, set persistent, restart systemd-journald
to apply:
[simterm]
root@bttrm-stage-console:/home/admin# systemctl restart systemd-journald
[/simterm]
Check the logs directory:
[simterm]
root@bttrm-stage-console:/home/admin# ll /var/log/journal/ total 4 drwxr-xr-x 2 root root 4096 Sep 24 17:54 7066493f616c4a6285c19cfaa2884b09 root@bttrm-stage-console:/home/admin# ll /var/log/journal/7066493f616c4a6285c19cfaa2884b09/ total 57348 -rw-r----- 1 root root 58720256 Sep 24 17:54 system.journal
[/simterm]
Reboot the EC2 and check logs again:
[simterm]
root@bttrm-stage-console:/home/admin# journalctl --list-boots -1 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:56:19 EEST 0 591136b94276490580f4e39e576e639e Tue 2019-09-24 17:56:20 EEST—Tue 2019-09-24 17:56:41 EEST
[/simterm]
Now you are able to check the previous reboot log:
[simterm]
root@bttrm-stage-console:/home/admin# journalctl -b -1 -- Logs begin at Tue 2019-09-24 12:13:02 EEST, end at Tue 2019-09-24 17:57:36 EEST. -- Sep 24 12:13:02 bttrm-stage-console CRON[19407]: pam_unix(cron:session): session closed for user me-v3
[/simterm]
Size can be limited by setting the SystemMaxUse
, see the documentation.
Done.