In this lesson, you will learn how to diagnose storage-bound performance degradation by distinguishing between network latency and physical disk throughput limitations.
When your application—particularly a database—experiences sluggish write performance, the culprit is often "I/O Wait." This happens when the CPU is idle because it is forced to wait for the storage subsystem to acknowledge a read or write operation. In a Kubernetes environment, this frequently occurs when a Persistent Volume (PV) reaches its provisioned IOPS (Input/Output Operations Per Second) or throughput cap.
Analyzing the Bottleneck
To identify if storage is the bottleneck, start by examining the iostat output on the node. Specifically, look for high %util and await values:
# Monitoring device latency on the host node
iostat -xz 1
await: The average time (in milliseconds) for I/O requests to be served. If this value is high, your application is blocked.%util: The percentage of CPU time during which I/O requests were issued to the device. If this is near 100%, your disk is saturated.
In Kubernetes, you can correlate these node-level metrics with Pod-level storage latency using tools like OpenTelemetry or Prometheus queries focusing on kubelet_volume_stats_available_bytes and volume latency histograms.
Distinguishing Throughput from IOPS
It is critical to distinguish between two common limits:
- IOPS Limits: You are performing too many small, random operations. The solution is often increasing the provisioned IOPS or moving to a higher-performance disk class (e.g., switching from standard SSDs to Provisioned IOPS SSDs).
- Throughput (Bandwidth) Limits: You are moving too much data at once. This hits the network-based speed cap between your compute node and your cloud storage provider.
If your metrics show high await times but low disk utilization, you may be hitting a storage-side rate limit or experiencing network congestion on the virtualized block storage link. Before scaling up disk size, ensure your database connection pools are tuned to avoid excessive small-block metadata writes, which often trigger these bottlenecks prematurely.
Key Takeaway: Use iostat to track await and %util on your host nodes to confirm if disk saturation—rather than application logic—is the source of your I/O wait latency.