Tracing User-space Stack Growth

In this lesson, you will learn how to monitor the stack usage of a user-space process to proactively detect stack overflow risks before they trigger a SIGSEGV.

The Problem of Stack Exhaustion

In memory-constrained systems or applications with deep recursion, the stack size is finite—typically defined by ulimit -s (often 8MB). If a process exceeds this limit, the kernel has no choice but to terminate it. Unlike heap allocations, which we can monitor via malloc hooks or brk/mmap tracing, stack growth is managed implicitly by the CPU’s stack pointer (RSP on x86_64).

Observability via eBPF

To observe stack growth, we don't look at the size of the stack at rest; we look at the movement of the stack pointer during function execution. The goal is to capture the "high-water mark" of stack usage for a specific thread.

We can use an eBPF program attached to a process to inspect the stack pointer value on every function entry. By comparing the current RSP to the base of the stack (found in /proc/[pid]/maps), we can derive the current stack depth:

SEC("uprobe/libc:__libc_start_main")
int trace_stack_depth(struct pt_regs *ctx) {
    u64 rsp = PT_REGS_SP(ctx);
    // Logic: Calculate distance between stack_base and current RSP
    // Update a per-thread map with the max observed displacement
    return 0;
}

Why This Matters

In deeply recursive algorithms, such as those found in parsers or graph traversal engines, stack usage is highly dynamic. If your application handles variable input sizes, an unusual input might push the recursion depth beyond your current stack allocation.

Monitoring this allows you to:

1. Identify "Spiky" Threads: Determine which threads are consuming more stack than anticipated.
2. Right-size Resources: Adjust thread-specific stack sizes using pthread_attr_setstacksize based on empirical evidence rather than guesswork.
3. Prevent Crashes: Set up alerts when the stack pointer approaches the guard page, allowing for graceful degradation instead of a hard crash.

Key Takeaway: Monitoring stack growth requires tracking the stack pointer's high-water mark via eBPF or instrumentation to prevent unexpected stack overflows in deeply recursive or memory-constrained environments.