Monitoring Disk I/O Queue Depths

In this lesson, you will learn how to correlate rising I/O wait times with increasing disk request queue depths to identify storage bottlenecks under heavy system load.

The Relationship Between Queues and Latency

When your application requests data from a disk, the Linux kernel places that request into a block layer queue. If the hardware can process requests faster than they arrive, the queue remains shallow, and latency stays low. However, under heavy load, requests accumulate in the queue while waiting for the storage device to become available.

"I/O Wait" (iowait) is a metric representing the percentage of time a CPU spends idling while waiting for outstanding block I/O requests to complete. When you see iowait climb in tools like top, it is a symptom; the underlying cause is often a saturated disk request queue.

Correlating Metrics with eBPF

To diagnose this, we must look at the queue depth—the number of requests currently held in the block layer—and map that to the time threads spend in an uninterruptible sleep state. Using BPF, we can hook into block_rq_issue and block_rq_complete to track the lifecycle of a request.

Consider a scenario where the queue depth grows as the disk controller reaches its IOPS limit. As the queue grows, the latency of each individual request increases because every request must wait for those ahead of it to flush. We can visualize this correlation with a simple trace:

// Pseudo-logic for tracking queue depth
SEC("tracepoint/block/block_rq_issue")
int count_issue(struct trace_event_raw_block_rq_issue *ctx) {
    // Increment per-device counter
    return 0;
}

SEC("tracepoint/block/block_rq_complete")
int count_complete(struct trace_event_raw_block_rq_complete *ctx) {
    // Decrement per-device counter
    return 0;
}

By observing these counters alongside iowait stats, you can confirm if your system is experiencing a "queue backup." If your queue depth increases steadily while iowait spikes, you are likely dealing with a physical device bottleneck (throughput saturation) rather than a misbehaving kernel driver or lock contention.

Key Takeaway: A high I/O wait percentage is almost always caused by a mounting backlog of requests in the block layer queue, indicating that the hardware cannot keep pace with the submission rate of your applications.