In this lesson, you will learn how to implement a watchdog mechanism to identify and recover from processes that have stopped working without crashing or exiting.
The Problem: The "Zombie" Process
Sometimes, a process doesn't "crash" in the traditional sense where the OS sends a SIGSEGV or SIGTERM. Instead, it enters a "silent failure" state: it remains in the process table as Running (or Sleep), but its internal logic has locked up. Perhaps a deadlock in a mutex (which we discussed in lesson 20) or a network thread that has permanently hung makes the process functionally dead even though the kernel sees it as alive.
The Watchdog Pattern
To detect this, we use a Watchdog. A watchdog is a secondary, lightweight process or a cron job that periodically checks the health of your primary application.
The simplest way to implement this is through a "heartbeat" file. Your application updates a timestamp in a file every few seconds. The watchdog checks if that timestamp is older than a specific threshold.
# A simple watchdog script (watchdog.sh)
THRESHOLD=30 # seconds
FILE="/tmp/app_heartbeat"
while true; do
if [ ! -f "$FILE" ] || [ $(($(date +%s) - $(stat -c %Y $FILE))) -gt $THRESHOLD ]; then
echo "Process hung. Restarting..."
killall my_app && ./my_app &
fi
sleep 10
done
Proactive Monitoring
While the heartbeat file works for logic hangs, you can also use procfs (lesson 13) to verify that the process is actually doing useful work. For example, check if the process is performing context switches (lesson 25) or system calls (lesson 17). If a process that should be active has zero context switches over a minute, it is almost certainly stuck.
By combining procfs metrics with a heartbeat check, you create a robust safety net that restarts failed services automatically, ensuring high availability even when the kernel itself doesn't realize the process has failed.
Key Takeaway: A watchdog process provides a vital external pulse check, using heartbeats and resource activity metrics to identify and recover from silent application hangs that the kernel alone cannot detect.