In this lesson, you will learn how to transform raw, aggregated eBPF stack trace data into human-readable FlameGraphs to pinpoint performance bottlenecks.
From Raw Traces to Visual Insights
When you collect stack traces using BPF_STACK_TRACE maps, you typically end up with a collection of call chains—lists of instruction pointers representing the state of the call stack at various points in time. While vital, these raw counts are impossible to read at scale. To make sense of them, we aggregate these traces into a format compatible with visualization tools like Brendan Gregg’s FlameGraph.
The core logic involves two steps: Symbolication and Aggregation.
- Symbolication: You must resolve instruction pointers into human-readable function names using the ELF headers of the binary or the kernel symbol table (
/proc/kallsyms). - Aggregation: You compress identical call stacks, counting how many times each unique stack appeared.
Creating the Input Data
To build a FlameGraph, you generate an intermediate "folded" format where each line is a semicolon-separated stack trace followed by a count:
kernel_entry;sys_read;vfs_read;ext4_file_read_iter 45
kernel_entry;sys_read;vfs_read;xfs_file_read_iter 12
In your custom observability CLI, you can implement this by iterating over your BPF_STACK_TRACE map. For every entry, you resolve the addresses and concatenate them. If you are profiling the scheduler or hot code paths, you can use a dictionary/map in your user-space agent to increment counters for identical stack signatures before piping the output to the flamegraph.pl script.
Why Custom Visualizations?
Standard metrics give you the what (e.g., latency is high), but custom FlameGraphs give you the why (e.g., a specific library call in the kernel's memory management path is stalling). By building these directly into your observability pipeline, you allow engineers to transition from a high-level alert directly into a visual representation of the kernel state at the moment the performance degradation occurred. This reduces "mean time to innocence" by visually isolating whether the bottleneck resides in user-land, the syscall interface, or deeper kernel internals.
Key Takeaway: Converting raw kernel stack traces into folded format enables you to visualize complex execution patterns, turning millions of granular events into a clear map of your system's performance hotspots.