Watchdog timer implementation

In this lesson, you will learn how to configure hardware and software watchdogs to ensure system recovery when a kernel hang occurs.

A watchdog is essentially a "dead man's switch" for your computer. It is a timer that expects a periodic signal from the operating system—a process known as "kicking the dog." If the system hangs, the kernel stops updating the timer, and upon expiration, the watchdog initiates a hard reset or a controlled reboot to restore availability.

Hardware vs. Software Watchdogs

• Hardware Watchdogs: These utilize dedicated physical circuitry on the motherboard or CPU. Because they operate independently of the main kernel execution flow, they are the most reliable way to recover from a complete kernel panic or a "hard lockup" where interrupts are no longer serviced.
• Software Watchdogs: These rely on the kernel’s timer interrupt infrastructure (often using jiffies or high-resolution timers). While easier to implement, if the kernel hangs deep within a spinlock or a non-maskable interrupt loop, the software watchdog may fail to trigger.

Implementation

To interact with the watchdog, Linux uses the /dev/watchdog character device. You typically need the watchdog daemon (watchdogd or similar) to manage the heartbeats.

To configure the software watchdog, you can load the softdog module:

# Load the software watchdog module
modprobe softdog

# The daemon then writes to the device to prevent expiration
cat > /dev/watchdog

If the system hangs, the hardware/software timer counts down to zero. In a production environment, you configure the timeout via /etc/watchdog.conf. You can also monitor specific parameters; for example, if the system load exceeds a threshold or a specific file isn't updated within a time window, the daemon can refrain from "kicking" the dog, forcing a reboot to clear a stalled state.

In mission-critical systems, pairing this with a hardware-based timer is the standard for high availability, as it ensures that even if the scheduler stops executing tasklets or handling interrupts, the system remains capable of returning to a functional state.

Key Takeaway: A watchdog timer provides a failsafe mechanism that forces a system reboot if the kernel fails to signal its health within a pre-defined window, bridging the gap between a non-responsive system and a recovered one.