In this lesson, you will learn how to design systemd automation services that ensure high availability by failing over gracefully between redundant nodes.
High availability in a systemd context isn’t about building a complex cluster from scratch; it is about leveraging systemd's ability to monitor state and respond to health changes. When you have two nodes performing the same task, the goal is to ensure that only one node is "Active" at a time, while the other remains in a "Standby" state, ready to take over if the first fails.
The Mechanism: Active/Passive Coordination
To achieve this, we use a combination of ConditionPathExists and a shared resource (like a virtual IP or a mounted lock file). A "heartbeat" service on each node monitors the health of its peer. If the primary service fails or the node goes offline, the standby node detects the change and triggers its own service.
Consider a simple standby service configuration that waits for a specific file to be absent before starting:
# /etc/systemd/system/app-ha.service
[Unit]
Description=High-Availability Automation Task
# Only run if the primary node's lock file does not exist
ConditionPathExists=!/var/run/shared/primary_is_active
[Service]
ExecStart=/usr/local/bin/sync-data.sh
Restart=on-failure
RestartSec=5
Implementing the Failover Logic
To make this resilient, you combine this with an ExecStartPre script that checks the status of the peer. If the peer is unreachable or the lock file is missing, the standby node promotes itself by creating the lock file. Because systemd handles the service lifecycle, you can use OnFailure triggers to execute cleanup scripts or notify administrators via D-Bus, ensuring that the "Active" status is strictly maintained.
By using ConditionPathExists in combination with external heartbeat monitors, you create a "wait-and-watch" pattern. This ensures your automated tasks don't suffer from "split-brain" scenarios where both nodes attempt to execute the same job simultaneously, which could lead to data corruption or service collisions.
Key Takeaway: High-availability in systemd is best achieved by using conditional units that gate service execution based on the presence of shared state, ensuring only one instance operates at a time.