Linux: processes core dumps, systemd-coredump and Debian

By | 03/14/2020

Need to get a dump from fro ma PHP process on Debian 9.

In this post will take a Linux kernel mechanism to create and manage processes dumps.

Kernel’s dumps are created in another way, check Kdump на Arch Wiki.

Linux Core Dump

The kernel will create a process dump if it performed an invalid operation and has to be stopped.

To do so, the kernel will send a special signal to such a process so the process can handle it itself or using standard mechanisms and will cause the kernel to apply the mechanism to create a dump of the memory of this process.

The complete list can be found in the kernel source in the SIG_KERNEL_COREDUMP_MASK macros:

...
#define SIG_KERNEL_COREDUMP_MASK (\
        rt_sigmask(SIGQUIT)   |  rt_sigmask(SIGILL)    | \
  rt_sigmask(SIGTRAP)   |  rt_sigmask(SIGABRT)   | \
        rt_sigmask(SIGFPE)    |  rt_sigmask(SIGSEGV)   | \
  rt_sigmask(SIGBUS)    |  rt_sigmask(SIGSYS)    | \
        rt_sigmask(SIGXCPU)   |  rt_sigmask(SIGXFSZ)   | \
  SIGEMT_MASK	
...

Which is used by the sig_kernel_coredump macros:

...
#define sig_kernel_coredump(sig)	siginmask(sig, SIG_KERNEL_COREDUMP_MASK)
...

Which is called in case of Fatal-errors and in its turn will call the do_coredump() function:

...
  fatal:
                ...
    if (sig_kernel_coredump(signr)) {
      if (print_fatal_signals)
        print_fatal_signal(ksig->info.si_signo);
      proc_coredump_connector(current);
                        ...
      do_coredump(&ksig->info);
    }
...

And do_coredump() will create a memory dump and will save it on a hard disk.

Signals and dump creation

Let’s check how it’s working.

Take a simple code in C:

#include <stdio.h>
#include <unistd.h>

int main() {

    while(1) {
        pid_t pid = getpid();
        printf ("Working with PID %lu\n", pid);
        sleep(5);
    }
}

Build and run:

[simterm]

$ gcc make_dump.c -o make_dump
$ ./make_dump 
Working with PID 2714790

[/simterm]

In another terminal – send a signal, for example SIGSEGV (Segmentation violation), code 11:

[simterm]

$ kill -s SIGSEGV 2714790

[/simterm]

In a terminal with the app running check its output:

[simterm]

$ ./make_dump 
Working with PID 2714790
...
Working with PID 2714790
Segmentation fault (core dumped)

[/simterm]

Check a dump file:

[simterm]

$ ls -l /tmp/ | grep 2714790
-rw------- 1 setevoy setevoy  380928 Mar 10 11:24 coredump-make_dump.2714790

[/simterm]

In the same way, you can create a dump for any already running process, for instances – run sleep:

[simterm]

$ sleep 100 &
[1] 2761144

$ kill -s SIGSEGV 2761144
[1]+  Segmentation fault      (core dumped) sleep 100

$ file /tmp/coredump-sleep.2761144 
/tmp/coredump-sleep.2761144: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'sleep 100', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/usr/bin/sleep', platform: 'x86_64'

[/simterm]

GDB – create a core dump

Besides of a signal sending you can use gcore from the gdb:

[simterm]

$ sleep 100 &
[1] 2762961

$ sudo gcore 2762961
...
Saved corefile core.2762961

$ file core.2762961
core.2762961: ELF 64-bit LSB core file

[/simterm]

GDB – read a core dump

To check a dump’s content you can use gdb and pass an executable as a first argument and a path to a dump file as a second argument:

[simterm]

$ gdb make_dump coredump-make_dump.2714790 
...
[New LWP 2714790]
Core was generated by `./make_dump'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f50d566e27e in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6
(gdb)

[/simterm]

kernel.core_pattern

During dump creation kernel will check the kernel.core_pattern parameter which determines how the dump will be handled.

Here you can specify a path and a filename with specificators or pass it external dumps handlers like systemd-coredump (рассмотрим ниже).

Check the Naming of core dump files documentation here>>>.

Most used options here are:

  • %e executable filename (without path prefix)
  • %p PID of dumped process, as seen in the PID namespace in which the process resides
  • %t time of dump, expressed as seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC)

Our file /tmp/coredump-make_dump.2714790, which we observed in the GDB above consists of the kernel.core_pattern = /tmp/coredump-%e.%p:

  1. /tmp catalog
  2. %e – file name started with the coredump + an executable name файла make_dump
  3. %p – and a killed process PID – 2714790

Also, instead of using the direct path and file name you can pass dump data via a pipe | to the /dev/null or to a handler like systemd-coredump.

limits.conf

Before creating a dump the kernel also will check soft and hard limits for the core in the /etc/security/limits.conf:

[simterm]

$ cat /etc/security/limits.conf | grep core
#        - core - limits the core file size (KB)
#*               soft    core            0

[/simterm]

Or check directly in your current environment:

[simterm]

$ ulimit -c
unlimited

[/simterm]

A running process limits can be found in its limits from the /proc:

[simterm]

$ ./make_dump 
Working with PID 2753034

[/simterm]

And limits:

[simterm]

$ cat /proc/2753034/limits 
Limit                     Soft Limit           Hard Limit           Units     
...  
Max core file size        unlimited            unlimited            bytes
...

[/simterm]

fs.suid_dumpable

Sometimes a dump can be not created if a process made a suid-operation for example by using the setuid() syscall.

In this case, the behavior is controlled by the fs.suid_dumpable:

[simterm]

$ sysctl fs.suid_dumpable
fs.suid_dumpable = 2

[/simterm]

It can accept 0, 1 or 2, see the man proc:

  • 0: (default) This provides the traditional (pre-Linux 2.6.13) behaviour. A core dump will NOT be produced for a process which has changed credentials (by calling seteuid or similar) or whose binary does not have read permission enabled.
  • 1: (“debug“) All processes dump core when possible.
  • 2: (“suidsafe“) Any binary which normally would not be dumped (see “0” above) is dumped readable by root only.

systemd-coredump

The systemd of course, has its own dumps handler – already mentioned systemd-coredump.

On Arch Linux is used by default, for Debian 9 can be installed with apt:

[simterm]

$ sudo apt -y install systemd-coredump

[/simterm]

Config file – /etc/systemd/coredump.conf.

After installation configure the kernel – send dumps via a pipe to the systemd-coredump:

[simterm]

root@bttrm-production-app-1:/home/admin#  echo '|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e' > /proc/sys/kernel/core_pattern

[/simterm]

Check it:

[simterm]

root@bttrm-production-app-1:/home/admin# cat /proc/sys/kernel/core_pattern
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e

[/simterm]

Create some dump:

[simterm]

root@bttrm-production-app-1:/home/admin# sleep 10 &
[1] 27117
root@bttrm-production-app-1:/home/admin# kill -s 11 27117
[1]+  Segmentation fault      (core dumped) sleep 10

[/simterm]

coredumpctl

To work with dumps managed with systemd-coredump the coredumpctl utility can be used:

[simterm]

root@bttrm-production-app-1:/home/admin# coredumpctl 
TIME                            PID   UID   GID SIG COREFILE EXE
Mon 2020-03-09 20:10:16 EET   20882     0     0  11 present  /home/admin/dump_test
...
Tue 2020-03-10 09:04:14 EET   16786  1003  1003  11 present  /usr/sbin/php-fpm7.4
Tue 2020-03-10 12:23:43 EET   27117     0     0  11 present  /bin/sleep

[/simterm]

In the SIG column, we can see a signal sent to the process killed, in those cases it were 11 == SIGSEGV.

Dump files are located at the /var/lib/systemd/coredump directory:

[simterm]

root@bttrm-production-app-1:/home/admin# ll /var/lib/systemd/coredump/
total 125376
-rw-r-----  1 root root    25208 Mar  9 20:10 core.dump_test.0.6bb23c691d354e9dbf4382d109a5c1d4.20882.1583777416000000000000.lz4
...
-rw-r-----  1 root root    33962 Mar 10 12:23 core.sleep.0.6bb23c691d354e9dbf4382d109a5c1d4.27117.1583835823000000000000.lz4

[/simterm]

To observe a dump’s content use info with some MATCH condition, for example, a PID:

[simterm]

root@bttrm-production-app-1:/home/admin# coredumpctl info 27117
           PID: 27117 (sleep)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2020-03-10 12:23:43 EET (3min 3s ago)
  Command Line: sleep 10
    Executable: /bin/sleep
...
      Hostname: bttrm-production-app-1
       Storage: /var/lib/systemd/coredump/core.sleep.0.6bb23c691d354e9dbf4382d109a5c1d4.27117.1583835823000000000000.lz4
       Message: Process 27117 (sleep) of user 0 dumped core.
                
                Stack trace of thread 27117:
                #0  0x00007fc1a8053270 __nanosleep (libc.so.6)
                #1  0x000056159fa6f91f n/a (sleep)
                #2  0x000056159fa6f700 n/a (sleep)
                #3  0x000056159fa6c9a4 n/a (sleep)
                #4  0x00007fc1a7fbb2e1 __libc_start_main (libc.so.6)
                #5  0x000056159fa6ca7a n/a (sleep)

[/simterm]

Debian – core dump is not created

Everything described above is working perfectly on two laptops with Arch Linux, but on my project’s servers with Debian 9 dumps are created in a usual way, without  systemd-coredump.

The core_pattern config:

[simterm]

root@bttrm-production-app-1:/home/admin# cat /proc/sys/kernel/core_pattern
/tmp/core-%e.%p

[/simterm]

Run a process, kill it with 11 signal, check /tmp  – no dumps here.

Run an app again:

[simterm]

root@bttrm-production-app-1:/home/admin# ./dump_test 
Working with PID 27954

[/simterm]

And check its limits:

[simterm]

root@bttrm-production-app-1:/home/admin# cat /proc/27954/limits 
Limit                     Soft Limit           Hard Limit           Units     
... 
Max core file size        0                    unlimited            bytes
...

[/simterm]

The Soft limit is set to the zero value, e.g. dumps are disabled at all as kernel can’t use more than 0 bytes for a dump file.

Set in to the unlimited:

[simterm]

root@bttrm-production-app-1:/home/admin# ulimit -S -c unlimited

[/simterm]

Run an app again, check limits now:

[simterm]

root@bttrm-production-app-1:/home/admin# ./dump_test 1>/dev/null &
[3] 28399
root@bttrm-production-app-1:/home/admin# cat /proc/$!/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes
...

[/simterm]

Create a dump:

[simterm]

root@bttrm-production-app-1:/home/admin# sleep 10 &
[4] 28613

root@bttrm-production-app-1:/home/admin# kill -s 11 $!

[/simterm]

Check it:

[simterm]

root@bttrm-production-app-1:/home/admin# ls -l /tmp/coredump-sleep.28613 
-rw------- 1 root root 380928 Mar 10 12:47 /tmp/coredump-sleep.28613

[/simterm]

Done.

Useful links