Graceful Shutdown of Worker Processes

In this lesson, you will learn how to prevent data loss by ensuring your background workers finish their current tasks before exiting the process.

When you deploy a new version of your application or scale down your infrastructure, the orchestrator (like Kubernetes or PM2) sends a SIGTERM signal to your Node.js processes. By default, Node.js terminates immediately. If a worker is in the middle of processing a long-running job—such as generating a PDF or updating a database record—that job is abruptly killed, often leaving your system in an inconsistent state.

Implementing the Shutdown Sequence

To handle this gracefully, you must intercept the termination signal and prevent the process from exiting until the worker has either finished its current task or safely returned it to the queue.

let isShuttingDown = false;
let activeJobs = 0;

process.on('SIGTERM', async () => {
  if (isShuttingDown) return;
  isShuttingDown = true;
  
  console.log('Shutdown signal received. Finishing active jobs...');
  
  // Stop pulling new jobs from the queue
  worker.pause();

  // Wait for existing jobs to complete
  if (activeJobs > 0) {
    await waitForJobsToFinish();
  }
  
  process.exit(0);
});

The Worker Lifecycle Strategy

The core logic relies on two distinct phases:

1. Drain the Ingest: Immediately stop the worker from fetching new tasks from your queue. This prevents the "work pileup" problem where a process trying to shut down keeps pulling more items it can't finish.
2. Await Completion: Maintain an activeJobs counter. Every time a job starts, increment it; when a job finishes, decrement it. Your shutdown handler should only call process.exit(0) once that counter hits zero.

If your worker cannot complete the job within a reasonable timeframe (e.g., 30 seconds), it is standard practice to attempt to requeue the current payload back into your persistent queue so another worker can pick it up later. This ensures that no work is permanently lost due to a redeployment.

Key Takeaway: A graceful shutdown requires you to stop accepting new work immediately upon receiving a signal, while providing a period of grace for active tasks to finalize or safely requeue before the process terminates.