Handling Transient Service Failures

Implement automated retries and restart logic for unstable automation tasks.

In this lesson, you will learn how to make your automated systemd services resilient against temporary glitches by configuring intelligent restart behavior.

When you automate tasks using systemd, things occasionally go wrong due to transient issues—a network timeout, a database temporarily locking, or a remote API being momentarily unreachable. Instead of failing permanently and requiring manual intervention, you can instruct systemd to "try again" automatically.

Configuring Resilience in Service Units

The core of managing transient failures lies within the [Service] section of your unit file. By defining specific restart policies, you can recover from non-fatal errors without human oversight.

Consider this snippet for a service that periodically fetches data:

[Service]
ExecStart=/usr/local/bin/sync-task.sh
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=60
StartLimitBurst=3

Here is how these directives work together to handle instability:

Restart=on-failure: This tells systemd to restart the process only if it exits with a non-zero status or is terminated by a signal. If the task finishes successfully, it stays stopped.
RestartSec=5s: This adds a "cool-down" period. It prevents your service from slamming a service that is currently rebooting or recovering, giving it five seconds to stabilize before the next attempt.
StartLimitIntervalSec and StartLimitBurst: These are your safety nets. They define a threshold (e.g., if the service fails more than 3 times within 60 seconds, stop trying). This prevents a "restart loop" that could consume excessive CPU or flood logs when a service is genuinely broken rather than just transiently failing.

Choosing the Right Strategy

While on-failure is the most common for automation, you might occasionally use always if a service must remain up regardless of how it exited. However, for periodic tasks or cron-replacement scripts, on-failure is the gold standard because it respects the intentional completion of your work while guarding against the unpredictable nature of external dependencies. By configuring these parameters, you transform your automation from a fragile script into a robust system component.

Key Takeaway: Use Restart=on-failure combined with a RestartSec delay to build resilience, but always define a StartLimitBurst to prevent infinite restart loops on permanent failures.