Liveness Probes for Deadlock Detection

In this lesson, you will learn how to design proactive liveness probes that detect event loop starvation to trigger automated container restarts.

The Problem: When the Loop Freezes

In Node.js, your application runs on a single-threaded event loop. If your code performs a heavy synchronous operation—like a massive JSON.parse on a multi-megabyte object or a long-running recursive loop—the event loop becomes blocked. When this happens, your server stops responding to network traffic, including standard health checks.

A simple GET /health endpoint that just returns { "status": "ok" } is insufficient here. If the event loop is blocked, that endpoint will time out, but your orchestrator (like Kubernetes) might not realize the process is effectively deadlocked; it might just assume the network is slow.

Monitoring "Heartbeat" Latency

To detect a deadlock, we need a "liveness probe" that measures the health of the event loop itself rather than just the application's ability to respond to HTTP requests. We do this by tracking the delay between when a setTimeout is scheduled and when it actually executes.

let lastHeartbeat = Date.now();

// Schedule a recurring check every 100ms
setInterval(() => {
  const now = Date.now();
  const lag = now - lastHeartbeat;

  if (lag > 1000) {
    console.error(`Event loop lag detected: ${lag}ms`);
    // This process is effectively deadlocked.
    // We mark our internal state as unhealthy.
  }
  lastHeartbeat = now;
}, 100);

// Our Liveness Probe endpoint
app.get('/health/liveness', (req, res) => {
  const isHealthy = (Date.now() - lastHeartbeat) < 1000;
  isHealthy ? res.status(200).send('OK') : res.status(503).send('Deadlocked');
});

Why this matters

By setting a threshold (e.g., 1000ms), you define a "point of no return." If the event loop is so congested that it can't execute a simple 100ms interval for over a second, the process is no longer reliable. When your health check returns a 503, the orchestrator identifies that the pod is in a zombie state and kills it. This triggers a clean restart, clearing the memory and the blocked loop, restoring availability to your users.

Key Takeaway: A robust liveness probe measures event loop latency to detect deadlocks, ensuring the orchestrator restarts the process before users experience prolonged downtime.