Course contentsShow
← Linux
Lesson 1625 of 1,81914. Observability and Performance TroubleshootingFree lesson

Stack Trace Aggregation

Reconstruct application call stacks from eBPF samples to visualize hot code paths.

In this lesson, you will learn how to reconstruct fragmented eBPF samples into human-readable call stacks to identify the "hottest" code paths in your application.

The Challenge of High-Frequency Sampling

When you use tools like perf or eBPF to profile a system, you are essentially taking "snapshots" of the instruction pointer at fixed intervals. If you capture 1,000 samples per second, you end up with 1,000 individual memory addresses. While accurate, this raw data is impossible for a human to parse. Stack trace aggregation is the process of walking back from these instruction pointers through the stack frames to reconstruct the full calling hierarchy.

How Aggregation Works

To visualize a "hot path," we need to link each snapshot to its parent function, then its grandparent, and so on. In eBPF, we use a specialized BPF map type called BPF_MAP_TYPE_STACK_TRACE. When an event triggers (like a timer or a scheduler event), the kernel uses the bpf_get_stackid() helper to traverse the stack and store a unique identifier for that specific call chain in the map.

If you are profiling a high-load function, your aggregation logic might look conceptually like this:

// Inside your BPF program
int sample_stack(struct pt_regs *ctx) {
    u32 stack_id = bpf_get_stackid(ctx, &stack_map, BPF_F_USER_STACK);
    // Increment the count for this specific stack trace in a hash map
    u64 *count = counts.lookup_or_init(&stack_id, &zero);
    (*count)++;
    return 0;
}

Visualizing the Result

Once the raw samples are aggregated into counts mapped to specific stack IDs, you can pass this data to a tool like a FlameGraph generator. Instead of seeing thousands of individual events, the aggregation turns them into a width-based visualization where the widest bars represent the paths where your CPU is spending the most time.

By grouping identical stack traces together, you filter out the noise and expose the underlying architecture of your application’s performance bottlenecks, distinguishing between functions that are slow due to their own logic versus those that are just called frequently by other paths.

Key Takeaway: Stack trace aggregation transforms granular, high-frequency execution snapshots into hierarchical call patterns, allowing you to identify exactly which code paths are consuming the most system resources.