Tracing Network Latency with BPF

In this lesson, you will learn how to identify kernel-level performance bottlenecks by measuring the precise time packets spend traversing various points in the networking stack.

Understanding Packet Latency

When you experience high network latency, the delay isn't always in the cable or the remote server. Often, the bottleneck is inside the Linux kernel itself—caused by complex firewall rules, buffer bloat, or inefficient context switching. eBPF allows us to attach probes to specific "hook points" within the kernel stack to calculate the latency between them without modifying the source code.

Identifying the Bottleneck

To trace latency, we use two BPF probes: one at the start of a function (entry) and one at its exit. By storing the packet’s timestamp in a BPF hash map keyed by a unique identifier (like the packet’s skb address or a tuple of IP/port), we can calculate the "delta" between these two points.

Consider a simplified scenario where we want to measure the time a packet spends in the TC (Traffic Control) ingress hook:

// BPF map to store the start time
BPF_HASH(start_times, u64, u64);

int trace_entry(struct __sk_buff *skb) {
    u64 id = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    start_times.update(&id, &ts);
    return 0;
}

int trace_exit(struct __sk_buff *skb) {
    u64 id = bpf_get_current_pid_tgid();
    u64 *start = start_times.lookup(&id);
    if (start) {
        u64 delta = bpf_ktime_get_ns() - *start;
        // Log or output the delta to user space
        bpf_trace_printk("Latency: %llu ns\n", delta);
    }
    return 0;
}

By placing these probes before and after specific kernel modules—such as the packet filtering logic or the routing table lookup—you can pinpoint exactly which part of the stack is slowing down your traffic. If the delta spikes during specific traffic loads, you have successfully isolated a performance bottleneck that traditional tools like ping or netstat would simply hide inside the "kernel time" black box.

Key Takeaway: By measuring the time delta between entry and exit points in the kernel using eBPF maps, you can isolate specific sub-systems causing network latency without affecting traffic flow.