In this lesson, you will learn how to leverage systemd’s built-in watchdog mechanism to automatically restart services that have entered a frozen or unresponsive state.
How the Watchdog Works
A watchdog acts as a "heartbeat" monitor between systemd and your service. Instead of systemd blindly assuming your application is healthy just because the process is running, the service must actively "ping" systemd at regular intervals. If that heartbeat stops—because the process is deadlocked, hanging, or caught in an infinite loop—systemd detects the silence and kills the process to restart it.
Configuring the Watchdog
To enable this feature, you must configure two parameters in your service’s [Service] unit file.
WatchdogSec=: This defines the maximum time allowed between heartbeats. If systemd does not receive a ping within this period, it considers the service unresponsive.Restart=on-failure: This ensures that once systemd kills the "hung" service, it triggers a restart.
Here is how you would configure a service unit:
[Service]
ExecStart=/usr/bin/my-app
# The app must send a heartbeat every 30 seconds or less
WatchdogSec=30s
Restart=on-failure
The Heartbeat Mechanism
How does the application send the "ping"? Systemd provides a simple notification interface. If you are writing a service, you can send this signal via the sd_notify function from the libsystemd library or simply by sending a WATCHDOG=1 message to the systemd notification socket.
When your application is healthy, it loops through its logic and sends this notification:
// Pseudo-code example of sending the heartbeat
sd_notify(0, "WATCHDOG=1");
If your application hangs, it will stop reaching this line of code. After 30 seconds of silence, systemd will log a warning, send a SIGABRT to the process, and immediately restart it based on your Restart= policy. This is far more reliable than generic health checks, as it verifies that your service is still actively executing its logic rather than just holding a PID in the process table.
Key Takeaway: Watchdog functionality ensures high availability by requiring services to prove they are alive, allowing systemd to automatically recover processes that have become unresponsive.