Analyzing Kernel Oops and Panic Logs

In this lesson, you will learn how to parse raw kernel crash reports to pinpoint the exact instruction or memory address that triggered a system failure.

When the kernel encounters an unrecoverable error, it generates a "Kernel Panic." If the error is recoverable but causes instability, it produces an "Oops." In an embedded system, these logs are your primary diagnostic tool because they act as a "black box" recording the state of the processor at the moment of failure.

Anatomy of a Stack Trace

The most critical part of an Oops log is the backtrace. It tells you the execution path the kernel took leading up to the crash. A typical trace looks like this:

CPU: 0 PID: 123 Comm: my_driver Not tainted 5.15.0 #1
PC is at my_faulty_function+0x28/0x40
LR is at my_driver_ioctl+0x54/0x80
...
[<c010a320>] (my_faulty_function) from [<c010a560>] (my_driver_ioctl+0x54/0x80)
[<c010a560>] (my_driver_ioctl) from [<c02b1f80>] (vfs_ioctl+0x2c/0x40)

Decoding the Failure

To interpret this, follow these three steps:

1. Identify the PC (Program Counter): The PC value tells you the exact memory address where the CPU faulted. The offset (e.g., +0x28) tells you how far into that specific function the error occurred.
2. Match to Symbols: Use your cross-compilation toolchain’s addr2line utility. If your kernel was compiled with debugging symbols, you can map the address to a source file and line number: arm-linux-gnueabihf-addr2line -e vmlinux c010a320
3. Examine the Registers: The logs include register dumps (R0, R1, SP, etc.). If R0 is 0x00000000, it is a strong indicator of a NULL pointer dereference—a classic source of instability in custom drivers.

Always remember that the kernel prints the "Call Trace" in reverse-chronological order. The bottom-most entry is the entry point (often a system call), while the top entry is the specific function where the illegal memory access or instruction execution occurred. By correlating the PC with your vmlinux file, you can move directly from a cryptic hex dump to the faulty line of C code.

Key Takeaway: A kernel oops is a snapshot of system failure; by mapping the Program Counter to your symbol file using addr2line, you can translate raw memory addresses into specific lines of source code to isolate the root cause.