Isolating Node-Level Failures Using System Logs

Correlate container crashes with underlying node-level kernel messages to distinguish between application bugs and host infrastructure instability.

In this lesson, you will learn how to distinguish between application-level bugs and host-level infrastructure instability by correlating container crash events with system-level kernel logs.

The Problem: When Logs Lie

When a container crashes with a non-zero exit code (like 137 for OOMKilled), the immediate reaction is to debug the application code or check memory limits. However, if your application is robust, the culprit might not be the code, but the underlying host. If the Linux kernel encounters a hardware fault, a disk I/O hang, or a memory pressure stall (PSI), it may terminate processes globally or force-restart the container runtime, leaving your application logs suspiciously blank at the moment of failure.

Correlating Events

To isolate these failures, you must correlate the Kubernetes Event timeline with the host's dmesg or /var/log/syslog. If a pod restarts exactly when the host kernel reports a Hardware Error or an EXT4-fs error, you have identified a node-level failure rather than a code defect.

You can observe these system-level signals by querying the logs on the node directly:

# Check for recent kernel-level panics or filesystem errors on the node
ssh node-01 "dmesg -T | grep -E 'Critical|Hardware|I/O|Memory'"

# Check for OOM killer activity outside of Kubernetes memory limits
ssh node-01 "grep -i 'killed process' /var/log/syslog"

Distinguishing the Cause

Application Bug: The application logs show an unhandled exception or a stack trace immediately before the crash. The node’s system logs are quiet during that timestamp.
Node Infrastructure Instability: The application logs simply stop or show a sudden disconnection (e.g., EOF). Simultaneously, the node’s dmesg reports hardware interrupts, disk write errors, or memory parity errors.

By establishing this correlation, you avoid "chasing ghosts" in your application code when the actual fix involves draining the node or replacing faulty hardware. This distinction is critical for maintaining high availability in production environments where infrastructure noise is inevitable.

Key Takeaway: Always verify the health of the underlying host kernel before assuming an application crash is a software-related bug.