In this lesson, you will learn how to detect and trace segmentation faults and invalid memory accesses in real-time to pinpoint the exact instruction causing a crash.
When a process attempts to access memory it does not own or violates its memory protections, the kernel sends a SIGSEGV (segmentation fault) signal. While logs often show "Segmentation fault," they rarely tell you why the access occurred. To observe these violations in real-time without recompiling, we use dynamic tracing via eBPF to hook into the kernel’s signal delivery mechanism.
Tracking Violations with eBPF
The kernel’s send_sig_info function is the gatekeeper for signals. We can attach a tracepoint to this function to monitor whenever a process receives a SIGSEGV. By filtering for SIGSEGV and capturing the instruction pointer (IP), we can identify the exact memory address that caused the fault.
Consider this minimal logic for a tracing script:
// Pseudo-logic for an eBPF hook on signal delivery
int kprobe__send_sig_info(struct pt_regs *ctx, int sig, struct task_struct *p) {
if (sig == SIGSEGV) {
u64 ip = get_instruction_pointer();
bpf_trace_printk("Segfault in %s at address %lx\n", p->comm, ip);
}
return 0;
}
Analyzing the Faulting State
Once you detect the violation, the challenge is mapping the instruction pointer back to your binary. You can cross-reference the reported instruction pointer with the binary’s memory map found in /proc/<pid>/maps. If the address resides in a region marked with ---p (no permissions) or falls outside the process’s valid range, you have found your culprit.
Unlike simple post-mortem core dump analysis, this real-time monitoring allows you to catch intermittent memory corruption in long-running services where the process state might be too volatile for a standard debugger to attach. By observing the SIGSEGV in the exact context of the thread's execution flow, you can distinguish between a null pointer dereference and a buffer overflow that corrupted the stack pointer.
Key Takeaway: By using eBPF to instrument send_sig_info, you can correlate real-time memory access violations with instruction pointers, allowing you to debug volatile crashes that standard logs and post-mortem dumps often miss.