Analyzing Scheduler Fairness metrics

In this lesson, you will learn how to verify if the Completely Fair Scheduler (CFS) is effectively balancing CPU time across different cgroups.

The CFS uses the concept of vruntime (virtual runtime) to track how long each task has executed relative to its weight. In a system managed by Control Groups (cgroups), the kernel enforces fairness not just between individual processes, but between groups of processes. If one cgroup is consistently denied its fair share of cycles, your application performance will suffer from unexpected jitter or starvation, even if total CPU usage appears low.

Inspecting cgroup fairness

To verify if your cgroup is receiving its intended allocation, you must examine the cpu.stat file located within the cgroup hierarchy. Specifically, we look for nr_throttled and throttled_time.

If you are using cgroup v2, navigate to your cgroup directory and inspect the controller stats:

cat /sys/fs/cgroup/<your_group>/cpu.stat

The output provides critical metrics:

• usage_usec: The total CPU time consumed by the group.
• nr_throttled: The number of times the group hit its cpu.max limit.
• throttled_usec: The total time (in microseconds) the group was prevented from executing because it exceeded its quota.

Interpreting the data

If nr_throttled is steadily increasing, the CFS is effectively "policing" your cgroup. This confirms that the scheduler is preventing the group from consuming more than its allocated period, which is exactly what fairness looks like from the kernel's perspective.

However, if you observe high throttled_usec values alongside application latency spikes, it indicates that your cpu.max quota is too restrictive for the workload's current concurrency demands. Unlike traditional process-level scheduling, where tasks vie for vruntime, cgroup-level fairness treats the entire group as a single entity with a fixed "budget." When that budget is spent, the group is forcibly removed from the runqueue, regardless of its internal priority.

By monitoring the ratio of throttled_usec against usage_usec, you can quantitatively determine if the scheduler is starving your cgroup or if the workload simply requires a higher weight or quota to meet its performance SLOs.

Key Takeaway: Scheduler fairness for cgroups is verified by monitoring nr_throttled and throttled_usec; high throttle counts indicate the CFS is enforcing resource limits, while performance degradation confirms the need for quota adjustments.