In this lesson, you will learn how to use kernel tracepoints to link a specific application write request to the actual latency experienced at the block device layer.
When an application performs a write, that data traverses several layers—the filesystem, the block I/O scheduler, and eventually the hardware driver. By the time a disk reports a "slow write," it is often difficult to tell if the delay was caused by the filesystem, contention in the block queue, or the physical storage device itself.
Kernel-level tracepoints provide "hooks" at these critical transition points. By instrumenting these hooks, we can observe the lifecycle of a single I/O request. We primarily focus on two events: block_rq_insert (when the request enters the block layer) and block_rq_complete (when the driver finishes the operation).
Practical Observation
We can utilize trace-cmd or the perf subsystem to capture these timestamps. If you were to trace the block layer, your output would look conceptually like this:
# Tracing requests for a specific block device
perf probe -a block_rq_insert
perf probe -a block_rq_complete
# Capture the trace data
perf record -e block:block_rq_insert -e block:block_rq_complete -a sleep 5
By comparing the timestamps of the insert event and the complete event for the same request ID, you derive the exact Block I/O Latency.
If you see a long gap between insert and complete, the latency is occurring either in the I/O scheduler queue or within the hardware/firmware of the drive. If the gap between the application calling write() and the block_rq_insert is large, the bottleneck is actually in the filesystem layer or kernel memory management, rather than the disk.
Using this technique allows you to differentiate between "my application is waiting for the disk to spin" and "my application is waiting for the kernel to process the metadata." It turns a black box of I/O performance into a measurable, traceable sequence of events that clearly identifies exactly where in the storage stack your performance is stalling.
Key Takeaway: By correlating block_rq_insert and block_rq_complete tracepoints, you can isolate whether storage latency originates from filesystem overhead or the physical device driver.