Monitoring Writeback Latency

In this lesson, you will learn how to quantify the performance overhead incurred when the kernel flushes dirty page caches from system memory to persistent storage.

The Mechanism of Writeback

When a process writes data to a file, the Linux kernel doesn't immediately commit that data to the disk. Instead, it marks the memory page as "dirty" in the page cache. To maintain data integrity and free up memory, the pdflush or bdi-writeback threads eventually flush these pages to the storage device.

The latency here is often hidden from the application, but it manifests as I/O pressure, increased disk utilization, and sometimes "stalls" in system responsiveness. Monitoring this process is essential for distinguishing between application-level I/O blocking and kernel-level maintenance overhead.

Observability with eBPF

Since writeback happens asynchronously in the background, you cannot observe it via standard system call tracing. Instead, we target the kernel's writeback_dirty_page and writeback_write_inode tracepoints. By measuring the delta between the initiation of the flush and the completion of the I/O submission, you can build a histogram of writeback latency.

Consider a simplified logic flow for an eBPF probe:

// Track the start of writeback for a specific inode
SEC("tracepoint/writeback/writeback_dirty_page")
int trace_writeback_start(struct trace_event_raw_writeback_dirty_page *ctx) {
    u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&start_map, &ctx->inode, &ts, BPF_ANY);
    return 0;
}

// Calculate latency upon completion
SEC("tracepoint/writeback/writeback_write_inode")
int trace_writeback_end(struct trace_event_raw_writeback_write_inode *ctx) {
    u64 *start_ts = bpf_map_lookup_elem(&start_map, &ctx->ino);
    if (start_ts) {
        u64 delta = bpf_ktime_get_ns() - *start_ts;
        bpf_hist_update(delta); // Aggregated into a latency histogram
        bpf_map_delete_elem(&start_map, &ctx->ino);
    }
    return 0;
}

Why it Matters

If your system experiences periodic spikes in iowait without a corresponding spike in application-triggered write() syscalls, you are likely hitting a writeback bottleneck. By monitoring this latency, you can identify if your storage controller is unable to keep up with the kernel’s background flushing cadence, which often necessitates tuning /proc/sys/vm/dirty_ratio or dirty_expire_centisecs to smooth out the throughput.

Key Takeaway: Monitoring writeback latency allows you to isolate background kernel maintenance stalls from application I/O performance issues, providing visibility into the "hidden" cost of cache synchronization.