Observing File System Latency

Map latency distributions to specific VFS operations to troubleshoot slow disk access times.

Observing File System Latency

In this lesson, you will learn how to map latency distributions directly to specific Virtual File System (VFS) operations to pinpoint whether application slowdowns stem from slow disk I/O or kernel-level contention.

Bridging the Gap: VFS to Disk

While profiling block I/O (lesson 13) tells you how long it takes for a request to reach the hardware, it often masks the complexities of the kernel's file system layer. By the time a request hits the block device, it has already passed through the VFS and the file system driver (e.g., ext4, xfs). To truly troubleshoot performance, you must measure the latency of VFS operations like vfs_read and vfs_write.

The challenge is that vfs_read can serve data from the page cache (extremely fast) or trigger a disk fetch (slow). By using eBPF to capture the time difference between the entry and return of these VFS functions, we can generate a histogram of latency distributions.

Implementation Logic

We attach a kprobe to vfs_read to record the start time, and a kretprobe to capture the end time. By storing the duration in a BPF hash map keyed by the calling process or the file descriptor, we can categorize latency. If you see a bimodal distribution—where most reads return in microseconds (cache hits) but a subset takes milliseconds—you have definitive proof that your application's "stalls" are disk-bound rather than logic-bound.

SEC("kprobe/vfs_read")
int BPF_KPROBE(trace_read_entry) {
    u64 pid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&start_times, &pid, &ts, BPF_ANY);
    return 0;
}

SEC("kretprobe/vfs_read")
int BPF_KRETPROBE(trace_read_exit) {
    // Calculate latency: now - start_time
    // Update a histogram map for visualization
    return 0;
}

By correlating these distributions with specific file paths or process IDs, you can distinguish between systemic disk congestion and isolated I/O requests that are blocking your critical execution path.

Key Takeaway: Measuring VFS latency allows you to isolate whether application performance degradation is due to page cache misses and subsequent disk I/O, or if the bottleneck lies within the file system management layer itself.