In this lesson, you will learn how to leverage perf events to perform granular CPU utilization analysis specifically scoped to containerized workloads.
While general monitoring tools provide high-level metrics, perf allows us to peek into the kernel to understand exactly how CPU cycles are consumed by processes trapped within specific namespaces. Since containers are just groups of processes, we can use perf to attach to the PID namespace or the specific process IDs associated with a container to pinpoint bottlenecks.
Sampling by PID
The most direct way to profile a container is by targeting the PIDs associated with it. If you know the PID of the main process running inside your container (the entry point), you can capture a stack trace of where it spends its time:
# Capture CPU cycles for a specific container PID at 99Hz for 10 seconds
perf record -p <PID> -g --call-graph dwarf sleep 10
By adding the -g (call-graph) flag, perf records the stack traces. This reveals if the container is bottlenecked by kernel-mode system calls (like frequent read/write operations due to I/O) or if it is burning cycles on user-space computation.
Identifying Bottlenecks
Once you have recorded the data, you can analyze it to find "hot paths"—functions that consume the majority of the CPU time. This is invaluable when a container is experiencing high CPU usage but throughput remains low. You might discover that the container is spinning on a specific locking mechanism or frequently invoking syscalls that involve kernel-space networking stacks (like those we studied in earlier lessons regarding Netfilter or VXLAN encapsulation).
To view the breakdown, use:
perf report
The output highlights the percentage of CPU time spent in specific functions. If you see high overhead in functions related to bridge or nf_conntrack, you know the bottleneck is within the networking stack rather than the application logic. Because these events are gathered per-PID, the data remains isolated to the specific container group, effectively ignoring noise from the host or other containers.
Key Takeaway: By targeting container-specific PIDs with perf record, you can generate precise call-graphs that differentiate between application-level computation and kernel-level overhead, allowing for surgical performance tuning within namespaces.