BiteSizedChunks.comLearn one small thing at a time.
Course contentsShow
← Databases & SQL
Lesson 1360 of 1,95312. Key-Value and Caching EnginesFree lesson

Distributed Task Scheduling

Coordinate recurring background tasks across nodes with master-election patterns.

In this lesson, you will learn how to ensure recurring background tasks run exactly once across a distributed system by using master-election patterns in your caching layer.

The Challenge of Distributed Timing

When you scale your application to multiple nodes, you often face a "split-brain" scheduling problem. If every server node runs the same background task (like a daily database cleanup or generating a system report), you risk data corruption, duplicate emails, or massive redundant resource consumption. To prevent this, we need a way for nodes to "elect" a master node responsible for the task.

Using Redis for Master Election

Since we are utilizing our key-value engine as a central coordination point, we can implement an atomic "leader election" using the SET command with expiration (TTL).

Instead of hard-coding which server runs a task, each node attempts to claim a "lock" on the task before executing it:

-- Attempt to acquire the lock for a specific task
-- NX means "only set if not exists", EX 60 sets a 60-second timeout
SET task_lock_key "node_id_01" NX EX 60
  1. Contention: Every node tries to run the command. Only one node will receive an OK response because of the NX (Not Exists) flag.
  2. Execution: The node that receives OK becomes the temporary leader and runs the task.
  3. Heartbeat: The leader periodically updates the TTL of the task_lock_key.
  4. Failover: If the leader crashes, the key expires after 60 seconds. The next node to successfully run the SET command automatically becomes the new leader.

Coordination vs. Scheduling

It is important to remember that the database or cache layer here acts as the coordinator, not the scheduler. Your nodes are responsible for checking the lock periodically. By decoupling the "intent" (the task) from the "executor" (the node), you ensure that your background workloads remain idempotent and safe, regardless of how many nodes are currently online. This prevents the "thundering herd" problem while maintaining high availability.

Key Takeaway: Use atomic conditional set operations (NX/EX) to elect a single node as the leader for a recurring task, ensuring that even in distributed environments, each job runs exactly once.