In this lesson, you will learn how to dissect kernel logs to pinpoint exactly why the Out-of-Memory (OOM) killer terminated a specific containerized process.
When a Linux system runs out of memory, the kernel invokes the OOM killer to reclaim resources. In a containerized environment, this decision isn't arbitrary; it is governed by the oom_score and oom_score_adj of processes within their respective namespaces. To debug these events, you must look at the kernel buffer (dmesg) or /var/log/messages.
Decoding the OOM Kill Log
When the OOM killer activates, it dumps a stack of metadata. You will typically see a block starting with Out of memory: Kill process. Within this log, the kernel provides a snapshot of the state:
[1234.567] Out of memory: Kill process 4567 (my-app) score 850 or sacrifice child
[1234.567] Killed process 4567 (my-app) total-vm:204800kB, anon-rss:150000kB, file-rss:0kB
score: This represents the likelihood of the process being killed. It is calculated based on the memory usage and theoom_score_adjvalue (which you can modify via/proc/<pid>/oom_score_adj).anon-rss: This indicates the amount of anonymous memory (heap/stack) the process was holding. High usage here is the primary driver for OOM selection.total-vm: This is the virtual memory mapping. Note that a large virtual memory footprint doesn't always trigger an OOM if the process hasn't actually touched (allocated) physical pages.
Analyzing Namespace Impact
In environments using multiple namespaces, the OOM killer also respects the memory limits enforced by cgroups. If you are analyzing a container, check the memory.oom_control file within the cgroup hierarchy. If the kernel killed a process, it will often log the task and its cgroup path. By matching the cgroup path to your container runtime logs, you can determine if the process hit a container-specific memory limit (cgroup OOM) or a global system limit. Always check if a process had an oom_score_adj of -1000, which effectively makes it "OOM-immune," potentially causing the kernel to kill a more critical process instead.
Key Takeaway: The OOM killer selects processes by evaluating their oom_score, which is a function of actual physical memory usage (RSS) modified by oom_score_adj; always cross-reference the dmesg output with cgroup hierarchy logs to determine if the death was caused by a system-wide or container-local memory pressure event.