In this lesson, you will learn how to diagnose invisible performance degradation by querying raw cgroup statistics directly from the Linux kernel.
While tools like Prometheus provide high-level metrics, they often aggregate data in ways that mask transient spikes. If your application feels "sluggish" despite kubectl top showing plenty of available CPU and RAM, you are likely hitting cgroup limits—the kernel’s way of enforcing the boundaries you defined in your deployment manifests.
CPU Throttling
Even if your pod’s CPU usage average is low, it can be throttled if it bursts and hits its cpu.cfs_quota_us limit within a single period. To detect this, look at the cpu.stat file inside the pod’s cgroup directory on the host:
# Inside the container or via node shell:
cat /sys/fs/cgroup/cpu/cpu.stat
Pay close attention to nr_throttled. If this number is increasing over time, your application is being penalized by the Completely Fair Scheduler. You aren't "out of CPU," but you are exceeding the allowed burst capacity, leading to latency spikes that are often invisible to standard application-level APM agents.
Memory Pressure
Memory issues are trickier. If a pod is nearing its memory limit, the kernel’s OOM killer might be looming, or worse, the application might be experiencing "direct reclaim" latency. Check the memory.stat file for pgmajfault (major page faults) or look for memory.failcnt:
cat /sys/fs/cgroup/memory/memory.failcnt
A non-zero value in memory.failcnt is a smoking gun. It means the kernel attempted to allocate memory for your process, hit the memory.limit_in_bytes ceiling, and failed. This leads to heavy disk I/O (swapping to cache) or sudden process termination, which usually manifests as mysterious 5xx errors or connection resets that standard metrics might label as "network blips."
By inspecting these kernel-level counters, you bypass the abstraction layer of the orchestration platform to see exactly how the host OS is treating your workloads.
Key Takeaway: When high-level monitoring shows "healthy" resources but performance remains erratic, check cpu.stat for throttling or memory.failcnt for allocation exhaustion to reveal the kernel's enforcement actions.