Diagnostic Analysis of Container Memory Limits

In this lesson, you will learn how to identify and resolve scenarios where the Linux OOM-killer terminates your .NET application due to container memory constraints.

Understanding the OOM-Killer Context

When you deploy a .NET application to a container orchestrator (like Kubernetes), the container runtime enforces a memory.limit_in_bytes via cgroups. Unlike a traditional VM, if your process exceeds this limit, the Linux kernel’s Out-Of-Memory (OOM) killer will forcibly terminate the process. Crucially, this often happens before the .NET Garbage Collector (GC) has a chance to trigger a full collection, leading to "random" crashes that appear as Exit Code 137.

Diagnostic Strategy

To diagnose this, you must differentiate between the .NET GC’s internal memory view and the container’s cgroup limits.

1. Check for Exit Code 137: In your orchestrator logs, an exit code of 137 indicates a SIGKILL from the kernel.
2. Examine Cgroup Metrics: Use kubectl get pods or inspect the /sys/fs/cgroup/memory/memory.usage_in_bytes file inside the container to see if your process was hitting the ceiling.
3. Analyze GC Behavior: Use dotnet-counters to monitor GC Heap Size versus Working Set. If the Working Set (which includes native memory, memory-mapped files, and code) is rising sharply while Heap Size remains stable, you likely have a native memory leak or excessive unmanaged allocations (e.g., via ArrayPool or P/Invoke) that are pushing you over the limit.

A Diagnostic Code Check

You can expose the container's memory perspective within your application to log warning states before the crash occurs:

// Simple check to log if we are approaching the limit
long memoryLimit = GetCgroupMemoryLimit(); // Reads /sys/fs/cgroup/memory/memory.limit_in_bytes
long currentUsage = Process.GetCurrentProcess().PrivateMemorySize64;

if (currentUsage > (memoryLimit * 0.9))
{
    _logger.LogWarning("Approaching container memory limit: {Usage}/{Limit}", currentUsage, memoryLimit);
}

By comparing your GC heap metrics against these container-specific limits, you can distinguish between "managed memory pressure" (which the GC can manage) and "container boundary pressure" (which requires either increasing the orchestrator limit or reducing your process’s unmanaged footprint).

Key Takeaway: The Linux OOM-killer operates independently of the .NET GC; always monitor your container's cgroup memory usage alongside your process working set to prevent abrupt process termination.