In this lesson, you will learn how to use eBPF tools to pinpoint exactly where your storage requests are experiencing delays within the Linux kernel.
Understanding Storage Latency
When you perform an I/O operation (like writing a file to your Btrfs pool), the request travels through several layers: the Virtual File System (VFS), the filesystem driver (Btrfs), the block layer, and finally the physical device driver. Sometimes, a "slow" disk isn't actually a hardware failure; it might be kernel-level congestion or a bottleneck in the block layer.
eBPF (Extended Berkeley Packet Filter) allows us to trace these requests as they pass through the kernel without needing to restart the system or modify the kernel source code.
Using biolatency
The most effective tool for this is biolatency, part of the bcc (BPF Compiler Collection) toolkit. It creates a histogram of your block I/O latency, allowing you to see if your requests are completing in milliseconds or microseconds.
To trace the latency of your storage devices, run the following command in your terminal:
# Trace disk I/O latency and output a histogram every 5 seconds
sudo biolatency -D 5
When you run this, you will see output that looks like this:
usecs : count distribution
0 -> 1 : 0 | |
2 -> 4 : 50 |********** |
8 -> 16 : 200 |****************************************|
16 -> 32 : 20 |**** |
Why this matters for Btrfs
Because you are familiar with Btrfs, this tool is vital for debugging performance. If you notice a high frequency of requests in the 100ms+ range, it might indicate that Btrfs is struggling with metadata intensive operations or that a specific RAID profile is causing heavy write amplification. By using biolatency, you can differentiate between a slow physical drive and a bottleneck caused by complex snapshot management or heavy background scrubs.
If the histogram shows a "bimodal" distribution (two distinct peaks), it often suggests that most requests are fast (hitting the cache), while a smaller subset is hitting the slow underlying media, helping you prioritize where to optimize your filesystem layout.
Key Takeaway: Use the eBPF biolatency tool to generate real-time histograms of storage request times, allowing you to isolate whether performance issues reside in the kernel's block layer or the underlying storage media.