Detecting Stalled Systemd Units

Detecting Stalled Systemd Units

In this lesson, you will learn how to leverage systemd’s built-in watchdog functionality to automatically detect and recover services that have entered a deadlocked or non-responsive state.

When a background process hangs, it often stops responding to external stimuli while remaining technically "active" in the eyes of systemd. Rather than relying on manual intervention, you can use WatchdogSec to enforce a heartbeat. By configuring this, you instruct systemd to expect a "keep-alive" signal from the service at regular intervals. If the service fails to send this signal within the defined window, systemd assumes the process is stalled and will take action, such as restarting the unit.

Implementing the Heartbeat

To enable this, you must configure your service unit file and ensure your application code supports the systemd watchdog protocol.

1. Update the Unit File: In your service definition, add the following parameters under the [Service] section:

   [Service]
   # The time interval the service must check in
   WatchdogSec=30s
   # The action to take when the check-in is missed
   WatchdogSignal=SIGABRT
   Restart=on-failure
   

2. The Application Handshake: Your application must notify systemd that it is still alive. In C or Python, you can send a WATCHDOG=1 signal via the sd_notify interface. If the application enters a deadlocked loop (e.g., waiting for a resource that never unlocks), it will stop sending these pings. Systemd will detect the silence after 30 seconds and automatically cycle the unit, clearing the deadlock without human oversight.

Why this matters

This approach is critical for high-availability automation. Instead of letting a stuck process consume memory or hold file descriptors indefinitely, the watchdog forces a clean state. It treats a "silent" service as a "failed" service, ensuring your automated workflows remain resilient even when individual components experience internal race conditions or logic errors.

Key Takeaway: WatchdogSec creates a mandatory heartbeat for your services, allowing systemd to automatically restart units that have stopped responding, effectively self-healing from internal deadlocks.