Lifecycle Management of Daemons

In this lesson, you will learn how to configure service restart policies and failure behaviors in systemd to ensure your daemons remain highly available, even when they crash.

Why Daemons Need Recovery

In a perfect world, a daemon would run forever. In reality, software crashes due to memory leaks, network timeouts, or unexpected input. As we’ve discussed in our exploration of PID 1 responsibilities and unit orchestration, systemd acts as the supervisor for your services. Rather than manually restarting a failed process, you can define "restart policies" within the unit file to let systemd handle the recovery automatically.

Configuring Restart Policies

You define these behaviors in the [Service] section of a unit file (e.g., /etc/systemd/system/myapp.service). The two most important directives are Restart and RestartSec.

Consider this configuration:

[Service]
ExecStart=/usr/bin/my-daemon
Restart=on-failure
RestartSec=5s

• Restart=on-failure: This tells systemd to only restart the service if it exits with a non-zero status, is terminated by a signal, or hits a timeout. If the service exits cleanly (code 0), systemd assumes you intentionally stopped it and leaves it alone.
• RestartSec=5s: This introduces a delay before the restart attempt. This is crucial for high availability; if your daemon crashes because it couldn't connect to a database, slamming the database with a restart attempt every millisecond might only worsen the issue. A short delay allows the environment to stabilize.

Failure Behaviors and Limits

To prevent a "crash loop" from consuming all system resources, you can also use StartLimitIntervalSec and StartLimitBurst. These settings define how many times a service is allowed to restart within a specific timeframe. If the service exceeds these limits (e.g., restarting 5 times in 60 seconds), systemd will stop trying to restart it, preventing the daemon from repeatedly flapping and taxing the CPU or logging subsystems.

By delegating these policies to systemd, you ensure your services are self-healing, reducing the manual burden of process lifecycle management.

Key Takeaway: Use Restart=on-failure combined with RestartSec to create self-healing services that recover from crashes automatically without overwhelming the system.