In this lesson, you will learn how to bridge the gap between application-level logs and system-level process events to diagnose why a process might be crashing or hanging.
When troubleshooting, you often have two separate streams of truth: the application’s log file (which says "Error: Database Connection Lost") and the system’s view of the process (which might say "Process terminated with SIGSEGV"). To perform effective root-cause analysis, you must synchronize these by tying them to specific points in the process lifecycle: startup, execution, and termination.
The Synchronization Challenge
The biggest hurdle is that a process ID (PID) can be reused after a process terminates. If your application logs timestamps but not PIDs, you might be looking at logs from a previous, finished instance of that process. To correlate these successfully, you should ensure your application logs include the PID, which you can verify against procfs (from lesson 20).
Concrete Illustration
Imagine an application that unexpectedly disappears. By checking dmesg or /var/log/syslog, you might see a kernel message indicating a segmentation fault. If your application logs also record the process start time or a unique UUID at launch, you can map the "Last Log Entry" in your app to the "System Exit Event" in the kernel logs.
You can manually inspect the lifecycle status using the procfs entry for your process:
# Check the start time and status of a specific PID
grep -E 'Name|State|Tgid|Pid' /proc/[PID]/status
If you see the process state is Z (Zombie), you know the process has finished execution but hasn't been reaped by its parent. Comparing the timestamp of the last "App Log" entry against the "System Time" when the process entered the Z state allows you to prove whether the application crashed due to an internal error or was killed by the system (e.g., OOM Killer).
By aligning the wall-clock time of your application’s final log line with the system kernel events, you can pinpoint exactly what the system was doing—such as CPU throttling or memory pressure—at the precise moment your process lifecycle ended.
Key Takeaway: Correlating logs requires anchoring application events to the process PID and kernel-level status changes to ensure your analysis reflects the current instance of the process rather than a legacy record.