Handling Hangs and Deadlocks

Use watchdog mechanisms in systemd to automatically recover services from unresponsive states.

In this lesson, you will learn how to configure systemd watchdog mechanisms to automatically detect and recover services that have stopped responding due to hangs or deadlocks.

The Watchdog Concept

A deadlock or a hang can leave a service running in a "zombie" or unresponsive state, where it consumes system resources but fails to process requests. While systemd can restart crashed services, it doesn’t automatically know if a process is "stuck" processing data.

The systemd watchdog acts as a heartbeat monitor. You configure the service to send "keep-alive" pings to systemd at specific intervals. If systemd stops receiving these pings within the allotted time, it concludes that the service has hung and will forcefully restart it.

Implementing the Heartbeat

To enable this, you must modify your unit file and ensure your application is built to support the sd_notify protocol.

Enable the watchdog in the Unit File: Add the following directives to your service unit under the [Service] section:
```
[Service]
WatchdogSec=30s
Restart=on-failure
```
WatchdogSec=30s tells systemd to expect a signal at least every 30 seconds.
Send the heartbeat from your application: Your application code must actively report its health. In a simple C program, you would use the sd_notify function:
```
// Inside your main event loop
sd_notify(0, "WATCHDOG=1");
```

Why this solves hangs

When your service enters a deadlock, the execution thread that sends the WATCHDOG=1 signal stops running. Because the heartbeat is missed, systemd will trigger the Restart logic defined in your unit file. This provides a safety net for critical services, ensuring that even if a process enters an infinite loop or blocks indefinitely on a resource, the system will clear the state and attempt a fresh start without manual intervention.

It is important to set WatchdogSec to a duration long enough to account for legitimate heavy processing, but short enough to be useful for your specific availability requirements.

Key Takeaway: By integrating the systemd watchdog protocol into your applications, you enable automated recovery from deadlocks by forcing a service restart whenever the process stops signaling its health.