In this lesson, you will learn how to quantify the performance gap between configured CPU quotas and the actual execution time experienced by your applications due to cgroup throttling.
The Throttling Mechanism
When you set cpu.cfs_quota_us in a cgroup, the kernel enforces a hard limit: if a process consumes its entire quota within a period (cpu.cfs_period_us), the kernel forcibly pauses the task until the next period begins. This "throttling" is often invisible to standard CPU usage tools (like top), which might show a process as "100% busy" without revealing that it was actually suspended for half that interval.
Measuring the Impact
To observe the true impact of these limits, we must look at the kernel's internal accounting, specifically the nr_throttled and throttled_time metrics exposed in cpu.stat.
nr_throttled tracks the number of times the cgroup was paused, while throttled_time reveals the total nanoseconds the process spent in a "runnable but blocked" state. By monitoring these over time, you can calculate the "throttling ratio":
# Example logic to monitor throttling via the cgroup v2 interface
cat /sys/fs/cgroup/my_app/cpu.stat | awk '
/nr_throttled/ { throttled = $2 }
/throttled_usec/ { time = $2 }
END { print "Throttles: " throttled ", Total Stall (us): " time }
'
Advanced Observability with eBPF
While cpu.stat provides a cumulative snapshot, it doesn't show when or why individual threads were throttled. To gain granular visibility, you can attach an eBPF probe to the sched_stat_runtime tracepoint or use runqlat to measure the time spent waiting in the runqueue. By correlating cgroup_id with the task’s scheduling state, you can detect if a process is missing its deadlines precisely because of the CFS quota, or if it is waiting on I/O or lock contention.
If nr_throttled is high, your application is likely hitting a hard wall. If nr_throttled is low but latency is high, you aren't being throttled—you are likely experiencing the lock contention or NUMA locality issues we covered in earlier lessons.
Key Takeaway: Throttling observability is the delta between perceived CPU usage and actual scheduled runtime; always correlate cpu.stat metrics with scheduler wait-time to distinguish between configuration limits and systemic bottlenecks.