Analyzing coredumps on Linux

Coredumps on Linux are controlled by the /proc/sys/kernel/core_pattern sysctl setting, RLIMIT_CORE (core file size) and RLIMIT_FSIZE (file size) resource limits.

The values and settings of these differ among typical distributions and how to set them may also differ depending on the circumstances:

• Ubuntu likes to configure core_pattern to pipe coredumps to apport.
• Systemd also supports acquiring, saving and processing of coredumps (see systemd-coredump(8)).
• For systemd services, you’ll have to set resource limits related to coredumps using systemd itself. In an interactive shell session, you may be able to use ulimit -c unlimited

For more information on these topics, I suggest starting with a read-through of core(5).

Once you have acquired a coredump, you’ll need to analyze it. The most common debugger used for this is the GNU debugger, gdb.

Inspecting coredumps requires having a copy of the binary which generated the segfault, with debug information available. This typically means the compiler and release toolchains must be instructured to keep debug symbols and not strip them out.

On Debian/Ubuntu, release binaries have debug information stripped out, but there’s -dbg packages available for most major software which include the same builds with debugging symbols still intact.

A strategy I recommend is to create a Virtual Machine or container image using the exact same distribution. For example, assuming the software that segfaulted was running on a Ubuntu 18.04 LTS machine, boot an instance of Ubuntu 18.04 locally, then install gdb there as well as the -dbg version of whichever software you’re trying to debug.

Transfer the coredump to that machine, then run gdb /path/to/binary /path/to/coredump. Issuing bt or bt full should then give you a backtrace with symbols correctly resolved.