Multi-Instance Failure Coordination

You will prevent a 'thundering herd' effect where all nodes restart simultaneously after an external dependency returns.

In this lesson, you will learn how to prevent a "thundering herd" effect by staggering service restarts after an external dependency recovers from an outage.

The Thundering Herd Problem

When a shared dependency—like a database or an authentication microservice—goes down, all your Node.js instances will typically lose their connection simultaneously. If you implement simple retry logic without coordination, every instance will try to reconnect or "wake up" at the exact same millisecond the dependency returns. This sudden spike in traffic can instantly crash the recovered dependency again, creating a catastrophic restart loop.

Implementing Jittered Backoff

To coordinate recovery across multiple instances, you must introduce randomized delays (often called "jitter") into your reconnection logic. Instead of every instance attempting to reconnect immediately, they should wait for a random interval within a defined range.

In Node.js, you can achieve this by calculating a delay based on the instance's identity or a random factor:

const attemptReconnect = async (retryCount) => {
  // Base delay of 1s, multiplied by attempt number
  const baseDelay = 1000 * retryCount;
  // Add jitter: random value between 0 and 1000ms
  const jitter = Math.random() * 1000;
  
  const totalDelay = baseDelay + jitter;
  
  await new Promise(resolve => setTimeout(resolve, totalDelay));
  
  try {
    await connectToDependency();
  } catch (err) {
    attemptReconnect(retryCount + 1);
  }
};

Why This Works

By adding jitter, you spread the reconnection attempts across a window of time rather than a single point in time. This smooths out the load on your dependencies, allowing them to scale back up gracefully.

When managing multiple instances, you can further refine this by checking the health status of other instances or using a shared lock (like Redis) if you need to ensure that only a specific percentage of nodes reconnect at any given time. However, for most distributed systems, simple exponential backoff with randomized jitter is the most effective way to prevent your own infrastructure from becoming the source of a secondary outage.

Key Takeaway: Never trigger simultaneous reconnects after a failure; use randomized jitter to spread the load and prevent a thundering herd from crashing your dependencies.