Analyzing 'OOM Killed' Events

You will distinguish between OS-level process kills and Node.js-level memory exceptions for accurate incident reporting.

In this lesson, you will learn how to distinguish between an OS-level "OOM Killed" event and a Node.js-level FATAL ERROR: Ineffective mark-compacts near heap limit exception to improve your incident reporting.

When investigating why a Node.js process vanished, engineers often conflate two distinct scenarios. Misidentifying the source leads to "fixing" the wrong layer, wasting time on GC tuning when the issue is actually resource-constrained infrastructure.

The OS-Level Kill

If your process is terminated by the Linux OOM Killer, your Node.js application will have zero logs regarding the exit. The OS kernel, noticing that the system is out of memory, selects the process with the highest oom_score and sends a SIGKILL (signal 9) immediately. The process cannot catch this signal or clean up state.

How to detect: Check your system logs (e.g., dmesg or /var/log/syslog). You will see an entry like: Out of memory: Kill process 1234 (node) score 800 or sacrifice child

The Node.js-Level Exception

Conversely, Node.js has an internal memory limit (the max-old-space-size). If the heap grows to this limit and the Garbage Collector (GC) cannot reclaim enough memory, the V8 engine itself throws a fatal error.

How to detect: You will find an entry in your application logs or stderr: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Distinguishing the Two

The key differentiator is the process exit code and the presence of error logs.

OS Kill: Exit code 137 (128 + 9). No FATAL ERROR string in application logs. This indicates your process is exceeding its cgroup or node-level memory limit, regardless of Node's internal heap settings.
V8 Exception: Exit code 1. The FATAL ERROR message is explicitly printed to stderr by V8 before the process terminates. This indicates your heap usage is exceeding your defined max-old-space-size configuration.

When creating incident reports, always check dmesg first to confirm if the kernel intervened; if the kernel is silent, focus your analysis on your heap usage and GC behavior.

Key Takeaway: If you find a FATAL ERROR in your logs, you are hitting V8 heap limits; if the logs are silent and the exit code is 137, your environment’s cgroup limits are actively killing your process.