In this lesson, you will learn how to visualize CPU usage using flame graphs to identify the specific code paths responsible for silent latency in your production containers.
When your Kubernetes pods experience "hidden" latency—where CPU usage is high but throughput remains low—traditional metrics like total CPU utilization often fail to tell the whole story. You know that the pod is busy, but you don't know what it is actually doing. Flame graphs solve this by aggregating stack traces captured over time, representing the entire call stack of your application as a visual map.
Visualizing the Bottleneck
A flame graph displays the call stack on the Y-axis and the percentage of CPU samples on the X-axis. Each "box" is a function call. The wider the box, the more CPU cycles that function (and its children) consumed.
To generate these in a Kubernetes environment, you typically use a sampling profiler (like async-profiler for Java, py-spy for Python, or perf for C++/Go) to capture data during a high-load period:
# Capture 30 seconds of stack traces from a running pod
kubectl exec <pod-name> -- /usr/bin/perf record -g -p <pid> -- sleep 30
# Generate the SVG visualization
kubectl exec <pod-name> -- /usr/bin/perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > latency.svg
Interpreting the Graph
When you open latency.svg in your browser, look for "plateaus"—wide, flat-topped bars that span a significant portion of the width. If you see a wide bar high up in the stack representing a specific JSON parsing library or a regex function, that is your primary latency candidate. Unlike standard logs, the flame graph shows you exactly which library is holding up the thread, even if that code isn't explicitly logging an error.
By focusing optimization efforts on these "wide peaks," you directly address the inefficient code paths that contribute to tail latency, preventing them from triggering resource contention or hitting your Vertical Pod Autoscaler limits unnecessarily.
Key Takeaway: Flame graphs turn opaque performance data into a visual hierarchy, allowing you to identify and prune the most expensive code paths causing production latency.