Fault-Tolerant Async Orchestration

In this lesson, you will learn how to design a "Supervisor" pattern that keeps your asynchronous background workers running even when they stall or hang unexpectedly.

The Problem: The "Silent Hang"

When you orchestrate multiple asynchronous tasks, a worker might stop responding due to an unhandled promise rejection or a logic deadlock. Because these tasks are background operations, your main application might not realize the worker has effectively "died," leading to a silent gap in your data processing.

The Supervisor Architecture

To fix this, we implement a Supervisor pattern. Instead of letting tasks run in isolation, we treat our workers as children of a monitoring process. The Supervisor maintains a heartbeat or a timestamp for each worker. If a worker fails to report progress within a set time, the Supervisor assumes the worker is stalled, kills the zombie process/task, and spawns a fresh one to take its place.

Consider this simplified conceptual logic:

const workers = new Map();

function startWorker(id) {
  const worker = {
    process: spawnTask(), // Your async task logic
    lastHeartbeat: Date.now()
  };
  workers.set(id, worker);
}

// The Supervisor heart-beat check
setInterval(() => {
  for (const [id, worker] of workers) {
    const isStalled = (Date.now() - worker.lastHeartbeat) > 5000;
    
    if (isStalled) {
      console.log(`Worker ${id} stalled. Restarting...`);
      worker.process.kill(); // Cleanup
      startWorker(id);       // Respawn
    }
  }
}, 1000);

Why this works

By decoupling the execution of the task from the lifecycle management, you create a self-healing system. The Supervisor doesn't need to know why the task stopped; it only needs to know that the task should be alive. If it isn't, the Supervisor forces a reset to a known good state. This approach ensures that your system remains resilient even when individual components encounter unpredictable failures that would otherwise halt the entire pipeline.

Key Takeaway: A Supervisor ensures fault tolerance by continuously monitoring worker health and force-restarting tasks that fall outside of expected activity thresholds.