In this lesson, you will learn how to use perf to gain visibility into the Linux kernel’s networking stack performance by correlating hardware events with specific software execution paths.
While eBPF is excellent for granular, programmable tracing, perf is the primary tool for analyzing where the CPU spends its cycles during high-volume packet processing. By utilizing hardware counters—such as cache misses and branch mispredictions—alongside kernel function sampling, you can pinpoint bottlenecks in the networking stack that are often invisible at the application layer.
Profiling the Kernel Stack
To analyze the network stack, we focus on the kernel functions involved in packet reception and transmission. A common approach is to record the kernel's execution while generating traffic, then inspect the call graphs.
Use the following command to record samples of the kernel stack during a network-heavy workload:
# Record kernel stack traces for 10 seconds
sudo perf record -a -g -e cycles:k sleep 10
The -a flag captures system-wide activity, -g enables call-graph recording (crucial for seeing the stack hierarchy), and -e cycles:k instructs perf to count CPU cycles spent specifically in kernel space.
Interpreting the Bottlenecks
Once recorded, you can view the output using perf report. Look for heavy hitters like netif_receive_skb or ip_rcv. If you see high percentages in these functions, you are looking at the overhead of the protocol stack itself.
If you notice significant "cycles" being spent in kfree_skb (packet drops) or high cache-miss counts in the net_rx_action function, it often indicates that the kernel is struggling with memory allocation or excessive lock contention under load. By comparing these hardware counters against the software call paths, you can determine if a performance hit is due to inefficient packet handling logic or simply a lack of hardware-level optimization (like missing RSS/Receive Side Scaling configurations).
This "outside-in" view complements the "inside-out" view provided by eBPF, giving you a holistic understanding of how your hardware interacts with the kernel's networking architecture.
Key Takeaway: Use perf to correlate hardware performance counters with kernel call stacks to identify precisely where the CPU is stalling during network packet processing.