System-Wide Process Locking

In this lesson, you will learn how to prevent multiple instances of a script from clashing when running across a network of different machines.

The Challenge of Distributed Coordination

When you run automated tasks across several servers, you often face a "race condition." Imagine you have a backup script running on three different hosts, all trying to write to the same central database file simultaneously. Without a system-wide lock, these processes will collide, causing corrupted data. Because these processes exist on different hosts, standard local file locking (like flock on a single machine) isn't enough; you need a mechanism that is visible to every participant in your distributed system.

Implementing a Distributed Lock

The simplest way to implement a system-wide lock for shell orchestration is by using a "lock file" located on a shared network drive (such as an NFS mount). The strategy follows a three-step pattern: Check, Create, and Cleanup.

Here is a basic pattern you can use in your scripts:

LOCK_FILE="/mnt/shared_storage/task.lock"

# Attempt to create the lock file atomically
if (set -o noclobber; echo "$$" > "$LOCK_FILE") 2> /dev/null; then
    echo "Lock acquired. Starting task..."
    # Perform your sensitive operations here
    rm -f "$LOCK_FILE"
else
    echo "Task is already running elsewhere. Exiting."
    exit 1
fi

Why This Works

The magic happens with set -o noclobber. This shell option tells the system to fail if a file already exists rather than overwriting it. By combining this with a redirection, the shell ensures that the creation of the file is an "atomic" operation—meaning it happens in one indivisible step. If two machines try to create the file at the exact same millisecond, the operating system will grant success to one and return an error to the other.

This ensures that only one host, on one process, ever enters the "critical section" of your code. Once the task finishes, the script removes the lock file, effectively "unlocking" the system for the next waiting process.

Key Takeaway: Use atomic file creation on shared storage to enforce a single-process execution policy across multiple networked hosts.