Table of Contents
Introduction
A toolchain upgrade for an OS is not simple, as it requires all modules, components to be recompiled, tested, and redeployed again. Despite its apparent complexity, it helps spot problems (a lot) and improves the OS over time.
A team was assigned a task of upgrading it. After an inevitable struggle, the toolchain was upgraded, and everything worked as expected until somebody stomped on an issue where GDB refused to analyze core dumps.
gdb-multiarch <binary> <core>yield next result:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /sandbox/var/test_app_pie.bin...
warning: Can't open file /var/test_app_pie.bin during file-backed mapping note processing
warning: Can't open file /usr/lib/libatomic.so.1.2.0 during file-backed mapping note processing
warning: Can't open file /usr/lib/libc.so.6 during file-backed mapping note processing
warning: Can't open file /usr/lib/libgcc_s.so.1 during file-backed mapping note processing
warning: Can't open file /usr/lib/libm.so.6 during file-backed mapping note processing
warning: Can't open file /usr/lib/libunwind.so.1.0 during file-backed mapping note processing
warning: Can't open file /usr/lib/libc++abi.so.1.0 during file-backed mapping note processing
warning: Can't open file /usr/lib/libc++.so.1.0 during file-backed mapping note processing
warning: Can't open file /usr/lib/ld-linux-armhf.so.3 during file-backed mapping note processing
warning: core file may not match specified executable file.
[New LWP 260]
Core was generated by `/var/test_app_pie.bin'.
Program terminated with signal SIGABRT, Aborted.
#0 0x76d00714 in ?? ()
(gdb) bt
#0 0x76d00714 in ?? ()
#1 0x76cff130 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) info sharedlibrary
No shared libraries loaded at this time.The main issue here is that GDB lacks information about shared libraries. Therefore, even if the correct sysroot is provided, it will not resolve the problem.
Analysis
A brief review revealed that the toolchain upgrade was the primary culprit behind the issue. Subsequent investigations narrowed down the problem to the PIE (position independent executable) option, which has been enabled by default in LLVM starting from version 15.
This helped confirm that GDB had a problem with core dumps originating from PIE binaries. There were no workarounds at this point, so the problem was severe.
I began debugging GDB. It appeared that the displacement was calculated incorrectly.
The function at blame is svr4_exec_displacement, which goes all the way down to the core_target::xfer_partial call that returns zero-initialized memory of a program header table for an unavailable mapping of test_app_pie.bin binary, and this zero-initialized data messes up the displacement calculation algorithm.
Incorrect displacement prevents GDB from mapping shared libraries, resulting in an empty list.
Consequently, when a user executes the info sharedlibrary command, it returns a blank line. Unfortunately, even after providing the correct sysroot, the issue persists.
Workaround:
- set correct path to sysroot so there're no mapping errors.
- it is important to reload coredump. (
core-file <path_to_core>)
Luckily, this change recently fixed the issue, which is included in the gdb-16.1 release.
If you are using a version of GDB prior to 16.1, follow the workaround mentioned above if you encounter the issue.
Conclusion
The latest version of GDB now works seamlessly with PIE binaries, allowing users to enjoy the same non-PIE experience when analyzing core dumps. The issue is not as severe as I thought, as there's a simple workaround; yes, it is inconvenient, but it is better than not being able to debug a crash.