In this lesson, you will learn how to perform a forensic analysis of a kernel panic using a crash dump to identify the exact line of code that triggered the system failure.
When the Linux kernel encounters a condition it cannot recover from—such as a null pointer dereference or a hardware-induced machine check exception—it triggers a kernel panic. To debug this post-mortem, we rely on kdump to capture the contents of memory at the moment of the crash into a file called vmcore.
Analyzing the Post-Mortem
To investigate these dumps, we use the crash utility. Once you have the vmcore and the corresponding vmlinux debug symbol file, you invoke it like this:
crash vmlinux vmcore
Inside the crash interactive prompt, your first priority is to identify the execution context that led to the panic. The bt (backtrace) command is your most powerful tool:
crash> bt
PID: 1234 TASK: ffff880037a54000 CPU: 3 COMMAND: "kworker/u16:2"
#0 [ffff88003d987d60] machine_kexec at ffffffff81065fdb
#1 [ffff88003d987dc0] crash_kexec at ffffffff8110b992
#2 [ffff88003d987e20] panic at ffffffff810a473f
#3 [ffff88003d987ea0] oops_end at ffffffff81014e7a
#4 [ffff88003d987ec0] no_context at ffffffff810486c4
The stack trace reveals the sequence of function calls. By inspecting the instruction pointer (RIP), you can correlate the faulting address to a specific line in the kernel source code using sym. If the fault is in a module, you might need to check if memory corruption occurred earlier by inspecting global kernel structures or active slab caches with kmem -s.
Pinpointing the Trigger
Often, the panic is not the root cause but a symptom of memory corruption that occurred seconds or minutes earlier. If the backtrace shows a "General Protection Fault" or "Page Fault," check the registers (rd) to see if a pointer held an invalid memory address. If you suspect memory corruption, cross-reference the faulting task with the slab allocator state to see if the object being accessed was already freed or modified out-of-bounds.
Key Takeaway: Use the crash utility to reconstruct the kernel stack trace, cross-referencing the faulting instruction pointer against the kernel binary to isolate the exact code path that failed.