Tracing Signals and Process Termination

In this lesson, you will learn how to pinpoint the exact origin of a SIGKILL or an unexpected process crash using BPF-based signal tracing.

The Mystery of the Silent Death

When a process receives a SIGKILL (signal 9), it cannot catch, block, or ignore it. The process simply vanishes. Traditional tools like dmesg or auditd might tell you a process died, but they rarely reveal who sent the signal or what triggered the termination. Because SIGKILL bypasses user-space signal handlers, you must intercept the signal at the kernel level using kprobes.

Tracing Signals with BPF

To observe signal delivery, we hook into the kernel function send_signal. By instrumenting this function, we can capture the source PID (the sender), the target PID (the victim), and the signal number.

Consider this logic for a BPF program:

int kprobe__send_signal(struct pt_regs *ctx, int sig, struct task_struct *t) {
    if (sig == SIGKILL) {
        u32 tgid = t->tgid;
        u32 sender_pid = bpf_get_current_pid_tgid() >> 32;
        bpf_trace_printk("SIGKILL sent to PID %d by PID %d\n", tgid, sender_pid);
    }
    return 0;
}

Analyzing Process Termination

While SIGKILL comes from external sources, other crashes (like SIGSEGV or SIGILL) are often self-inflicted due to memory corruption. You can correlate signal delivery with the task's exit state. By attaching a kprobe to do_exit, you can record the termination status. If you track the signal delivery alongside the exit code, you can build a complete forensic timeline: first, the signal is delivered; milliseconds later, the process invokes do_exit.

By using bpf_get_stackid within these probes, you can capture the exact instruction pointer that triggered the signal, allowing you to differentiate between an administrative intervention (e.g., an OOM killer or a manual kill -9) and a legitimate application software fault. This is far more precise than checking process return codes after the fact, as it captures the transient intent of the kernel or the offending peer process in real-time.

Key Takeaway: By instrumenting send_signal and do_exit with BPF, you can distinguish between external process termination and internal runtime crashes, identifying the specific sender PID and stack trace for every fatal signal.