In this lesson, you will learn how to capture granular network protocol data as packets traverse the complex layers of the Linux network stack.
Bridging the Gap: Tracing vs. Sniffing
While traditional tools like tcpdump (using libpcap) intercept packets at the network interface layer (Layer 2), they often miss what happens deeper inside the kernel. When diagnosing issues like dropped packets, routing loops, or buffer bloat, you need visibility into how the kernel handles a packet as it transitions through the qdisc, IP routing, and netfilter hooks.
Tracing with kprobes and Tracepoints
To trace the internal journey of a packet, we utilize the instrumentation techniques we’ve already mastered: kprobes and tracepoints. By attaching a BPF program to specific kernel functions like netif_receive_skb (entry) or ip_rcv (IP layer processing), we can inspect the sk_buff structure directly.
The sk_buff (socket buffer) is the backbone of the Linux networking stack. It contains metadata about the packet, including its source, destination, device, and protocol information.
Practical Application: Identifying Latency
If you suspect the kernel is adding unexpected latency to your packets, you can write a BPF script to measure the "delta" between entry and exit functions:
// Example: Attaching a kprobe to measure packet traversal
int kprobe__netif_receive_skb(struct pt_regs *ctx, struct sk_buff *skb) {
u64 ts = bpf_ktime_get_ns();
bpf_map_update_elem(&start_times, &skb, &ts, BPF_ANY);
return 0;
}
By storing the timestamp in a BPF Map when the packet enters the stack and calculating the difference when it hits the socket layer or a transmit function, you can pinpoint exactly where in the stack your packet is being delayed. This level of granularity allows you to distinguish between delays caused by the hardware (NIC) versus those caused by kernel-side contention or filter rules.
By moving from passive packet sniffing to active kernel instrumentation, you gain the ability to "see" the internal state of the networking stack rather than just the raw traffic.
Key Takeaway: Packet tracing allows you to leverage BPF to inspect the sk_buff metadata at any transition point within the kernel, providing visibility into internal processing delays that standard network tools ignore.