In this lesson, you will learn how to quantify the time processes spend waiting in the scheduler runqueue to pinpoint CPU starvation bottlenecks.
The Scheduler Runqueue Bottleneck
Even when a CPU appears 100% busy, the most critical metric for performance isn't just utilization—it is scheduler latency. This is the duration a process spends in a "Runnable" state, waiting for the Completely Fair Scheduler (CFS) to grant it access to a physical CPU core. If your tasks are "runnable" but not "running," they are being starved, usually due to overcommitment or aggressive CPU throttling (like CFS quotas).
Observing Wait Time with runqlat
To diagnose this, we look at the time spent in the runqueue. While top shows us CPU percentages, it fails to show the queue length or the time spent waiting in that queue. We use the eBPF-based tool runqlat (from the BCC toolkit) to measure this latency distribution:
# Measure runqueue latency for all processes, printing histograms every second
/usr/share/bcc/tools/runqlat 1
The output provides a histogram of latency in milliseconds. You will see columns for the time range (e.g., 1ms to 2ms) and the count of tasks that experienced that specific wait time.
Interpreting the Results
- Low Latency (Microseconds): Your application is effectively gaining access to the CPU as soon as it wakes up.
- High Latency (Milliseconds/Seconds): If you see a large distribution in the higher buckets, your system is suffering from resource contention. This is common in containerized environments where CFS quotas force a process to wait for a new quota period, even if the host CPU is idle.
By correlating these histograms with your application’s throughput, you can differentiate between code-level bottlenecks and infrastructure-level starvation. If you see high latency without high total CPU utilization, check your cgroups CPU limits or verify if background tasks (like heavy I/O interrupts) are stealing cycles from your critical paths.
Key Takeaway: Scheduler latency is the invisible gap between a task becoming ready and actually executing; if this time increases, your application will stutter regardless of how much CPU capacity you have "provisioned."