Implementing Distributed Locks

In this lesson, you will learn how to ensure mutual exclusion across multiple background worker instances using a shared Redis-based distributed lock.

The Challenge of Multi-Instance Coordination

When you scale your background workers horizontally to handle high-throughput processing, you often run into a classic "split-brain" problem. If your workers process recurring tasks—like generating financial reports or cleaning up expired sessions—you need to guarantee that only one instance performs the operation at any given time. Traditional lock statements or SemaphoreSlim instances are memory-bound to a single process and will not work across separate containers or virtual machines.

Implementing the Lock with Redis

To coordinate across instances, we move the locking state to a shared infrastructure component like Redis. The core principle is an "atomic acquisition" pattern. We attempt to set a key with an expiration time (TTL); if the key already exists, the worker skips the task. If it succeeds, the worker holds the lock, executes the work, and then releases it.

Using a library like StackExchange.Redis, you would implement the lock acquisition logic like this:

public async Task<bool> TryExecuteLocked(string lockKey, TimeSpan ttl)
{
    // Atomically set the key if it doesn't exist
    var acquired = await _db.StringSetAsync(lockKey, "locked", ttl, When.NotExists);
    
    if (!acquired) return false;

    try {
        await DoWorkAsync();
    }
    finally {
        await _db.KeyDeleteAsync(lockKey);
    }
    return true;
}

Critical Considerations

When implementing this in a distributed environment, you must account for clock drift and process crashes.

1. TTL Safety: Always set a Time-to-Live (TTL) on your lock. If your worker instance crashes mid-execution, a lock without an expiration would stay held forever, deadlocking the entire system.
2. Fencing Tokens: In advanced scenarios, consider using a version counter or "fencing token" to ensure that if a lock expires while a slow process is still running, the process cannot accidentally complete its write operation, as it no longer holds the valid lock.

Key Takeaway: Distributed locks provide global mutual exclusion by offloading state to a shared external store, requiring atomic acquisition, TTL-based safety nets, and robust cleanup in finally blocks.