Profiling Block I/O Requests

In this lesson, you will learn how to trace the complete lifecycle of a block I/O request as it traverses the Linux storage stack from the user-space buffer to the physical disk controller.

The Lifecycle of an I/O Request

When an application issues a write or read syscall, the request does not simply "hit" the disk. It flows through a sophisticated pipeline designed to optimize throughput and ensure data integrity. Understanding this path is vital for diagnosing storage latency:

1. VFS & File System Layer: The syscall enters the Virtual File System, which translates the request into a set of blocks specific to the file system (e.g., ext4, xfs).
2. Generic Block Layer: Here, the request is turned into a struct request and placed into a per-device request queue. This is where I/O schedulers (like mq-deadline or kyber) determine the order of operations.
3. Block Driver & Controller: The request is converted into commands the hardware understands (like NVMe or SCSI commands) and handed off to the device driver, which communicates with the physical storage controller.

Observing the Journey with BPF

While perf can monitor block tracepoints, eBPF allows us to measure latency at each of these stages by calculating the time delta between entry and exit points. Using BCC tools like biolatency or biotop, you can observe the "latency histogram" to identify where requests are stalling—whether it is queuing delay in the block layer or the physical latency of the disk itself.

Consider this conceptual flow trace:

// Simplified logic for calculating request latency
kprobe:blk_account_io_start {
    // Record start time for request 'req'
    start_time[req] = bpf_ktime_get_ns();
}

kprobe:blk_account_io_done {
    // Record end time and calculate delta
    delta = bpf_ktime_get_ns() - start_time[req];
    // Push delta to a BPF histogram map
    hist.increment(bpf_log2l(delta));
}

By focusing on the transition between these layers, you can differentiate between application-side bottlenecks (too many small I/O operations) and underlying hardware saturation.

Key Takeaway: Profiling block I/O requires tracking the time a request spends in the request queue versus the time spent waiting for the physical hardware to acknowledge completion.