Implementing Watchdog Timers

In this lesson, you will learn how to use a "watchdog timer" to detect and recover from tasks that have silently stalled or entered a "zombie" state.

The Problem: The Silent Failure

In distributed task queues, a worker might pick up a job and then hit a resource deadlock, an infinite loop, or a network partition that prevents it from finishing. Unlike an explicit error, the system doesn't know the job failed; it just stays "in-progress" forever. If your system relies on processing every message, these stalled jobs become "ghost" tasks that block progress and consume concurrency slots.

The Watchdog Solution

A watchdog timer acts as an external supervisor. When a task starts, you register a timer that expects a "heartbeat" or a completion signal within a specific window. If the timer expires before receiving that signal, the watchdog assumes the worker has stalled and triggers a recovery routine—such as re-queuing the task, killing the unresponsive worker process, or alerting an administrator.

Conceptual Implementation

You can implement a basic watchdog using setTimeout within your task processor:

async function processTask(task) {
  // Set the watchdog: if task takes > 30s, force recovery
  const watchdog = setTimeout(() => {
    console.error(`Task ${task.id} stalled! Initiating recovery.`);
    handleStall(task);
  }, 30000);

  try {
    await performWork(task);
    clearTimeout(watchdog); // Task finished; stop the timer
  } catch (err) {
    clearTimeout(watchdog);
    throw err;
  }
}

Why it Matters

Without a watchdog, your system is vulnerable to "silent hang-ups." By enforcing a deadline, you shift the system state from "unknown" to "failed," allowing you to trigger automated recovery strategies like retrying the task on a different consumer. This pattern is essential for long-running workflows where you cannot rely on TCP timeouts alone to signal that a worker has lost its connection or stopped executing code.

Key Takeaway: A watchdog timer creates a "failsafe" deadline for long-running tasks, ensuring that silent stalls are treated as active failures that can be automatically recovered.