Managing Distributed State for Scheduled Jobs

Use distributed databases or shared locks to ensure job uniqueness in a cluster.

In this lesson, you will learn how to synchronize scheduled tasks across a multi-node cluster to prevent duplicate execution.

When you scale your automation from a single server to a cluster, systemd timers run independently on every node. If you have a task like "generate daily billing reports," you don’t want five servers performing the same database operation simultaneously. To solve this, you need a way to manage distributed state, ensuring that only one node acts as the "leader" or "owner" of a job at any given time.

Using Distributed Locks

The most reliable way to handle this is via a distributed lock, often managed by a tool like etcd, Consul, or even a shared database like Redis or PostgreSQL. Before your script performs its core logic, it must "claim" a lock.

Consider this pseudo-script logic that uses a distributed lock stored in Redis (via redis-cli):

#!/bin/bash
LOCK_NAME="billing_job_lock"

# Attempt to acquire a lock with a 300s TTL (Time-To-Live)
if redis-cli set "$LOCK_NAME" "locked" NX EX 300 > /dev/null; then
    echo "Lock acquired. Running job..."
    ./perform_billing_report.sh
    redis-cli del "$LOCK_NAME"
else
    echo "Job already running on another node. Exiting."
    exit 0
fi

Why This Matters

In this setup, the first node to reach the scheduler tries to write a key to the shared database. The NX (Not eXists) flag ensures that the operation only succeeds if the key doesn't already exist. If it succeeds, the node gains exclusive rights to run the task. If it fails, the other nodes simply exit, effectively deferring to the leader.

Using a Time-To-Live (EX 300) is critical. If a node crashes while holding the lock, the TTL ensures the lock automatically expires after five minutes, allowing the job to resume on another node during the next scheduled cycle. This pattern transforms your distributed environment from a chaotic race into a coordinated, serialized queue.

Key Takeaway: Use a distributed lock with an expiration timer in a shared data store to ensure that concurrent scheduled jobs execute exactly once across your entire cluster.