In this lesson, you will learn how to identify and quantify "accounting drift" between kernel-level cgroup resource reporting and application-level telemetry.
When running workloads in containers, you often monitor resource usage from two perspectives: the kernel’s view (via cgroup controllers like memory.current or cpu.stat) and the application’s internal metrics (such as a JVM’s heap usage or a database’s buffer pool size). Advanced engineers often encounter "accounting drift"—a discrepancy where the sum of application-reported usage is significantly lower than the cgroup-reported usage.
The Source of Drift
This drift typically arises because the kernel tracks resources at a page-level granularity, while applications track objects. The cgroup accounting includes kernel-level overhead that the application is often blind to, such as:
- Page Cache: Filesystem data cached by the kernel on behalf of the process.
- Kernel Structures: Memory allocated for kernel data structures (dentries, inodes) associated with the process’s file descriptors.
- Fragmentation: Internal memory fragmentation that the kernel tracks as "in-use" while the application views it as available within its allocated pools.
Quantifying the Discrepancy
To detect this, you must correlate high-frequency snapshots. If your application reports 1GB of heap usage, but cgroup.memory.current shows 1.4GB, you can use perf events or eBPF programs to inspect where the "hidden" 400MB resides.
Consider this simplified logic for tracking drift:
# Snapshot the cgroup usage
CG_MEM=$(cat /sys/fs/cgroup/container_id/memory.current)
# Query app telemetry (e.g., via local stats socket)
APP_MEM=$(curl -s localhost:8080/metrics | grep 'heap_usage_bytes')
# Calculate the drift
DRIFT=$((CG_MEM - APP_MEM))
echo "Accounting drift: $((DRIFT / 1024 / 1024)) MB"
If the drift is static, it is usually overhead. If the drift increases over time, you are likely looking at a memory leak in kernel-managed resources, such as a dentry leak where the application holds many ephemeral file handles that the kernel is failing to reclaim immediately. Always reconcile these two data sources before assuming an application memory leak; the truth often lies in the gap between the two.
Key Takeaway: Accounting drift is a primary diagnostic tool; when cgroup metrics exceed application reports, focus your investigation on kernel-managed buffers, page caches, and underlying slab allocations rather than just the application's internal heap.