Kernel Debugging with KGDB and KDB

In this lesson, you will learn how to leverage KGDB and KDB to pause kernel execution and inspect internal state directly over a serial connection.

When a kernel module crashes or behaves unexpectedly, printk logging is often insufficient because the system may hang before output hits the console. To solve this, we use KDB (a built-in command-line debugger) and KGDB (the remote GDB stub for the kernel).

The Architecture of Kernel Debugging

Think of this as a two-machine setup: a Target (the embedded device running your code) and a Host (your development workstation).

1. KDB: This is the "on-the-fly" debugger. It allows you to enter a command shell directly on the target's serial console when the kernel hits a breakpoint or panic. You can inspect registers, view memory, and peek at the stack trace without needing a connection to another PC.
2. KGDB: This acts as a GDB server. By passing kgdboc=ttyS0,115200 to your kernel boot arguments, you tell the kernel to export its state to GDB running on your host machine.

Practical Workflow

To enter the debugger, you can force a break by echoing to the sysfs trigger:

echo g > /proc/sysrq-trigger

Once triggered, the target system freezes. If you are using KGDB, your host-side GDB will catch this event. You connect using:

(gdb) target remote /dev/ttyUSB0

Once connected, you have full visibility into the kernel’s memory. You can inspect your module’s data structures:

(gdb) print my_module_data
(gdb) list *(my_function+0x10)

Because the kernel is paused, you can step through line-by-line exactly as you would with a user-space application, allowing you to catch race conditions or null pointer dereferences that are otherwise invisible to standard logging.

Key Takeaway: KGDB and KDB transform your embedded serial console into a powerful interactive workbench, allowing you to halt the entire kernel to inspect deep-level system states that printk simply cannot reach.