Course contentsShow
← System Design
Lesson 1866 of 1,91944. Real-World System: Web CrawlerFree lesson

Worker Health Monitoring and Failover

Implementing heartbeat mechanisms and reassigning URLs from failed workers to healthy ones.

Worker Health Monitoring and Failover

What You'll Learn

You'll learn how to detect when crawler workers crash or stall and automatically reassign their in-progress URLs to healthy workers.

The Problem

In a distributed web crawler with many workers, machines fail, processes crash, or workers hang. Without monitoring, URLs assigned to dead workers would never get crawled, creating black holes in your crawl coverage.

Core Mechanism: Heartbeats

Each worker sends periodic heartbeat signals to the coordinator—simple "I'm alive" messages every 10-30 seconds. Think of it like a pulse oximeter: if the beeps stop, something's wrong.

The coordinator tracks the last heartbeat timestamp for each worker. If a worker misses multiple consecutive heartbeats (typically 2-3 cycles), the coordinator marks it as unhealthy or dead.

Reassignment Strategy

When a worker is declared failed:

  1. Retrieve pending URLs: The coordinator looks up which URLs were assigned to that worker (often stored in a "worker_id → URL_list" mapping in Redis or a database)
  2. Return to frontier: Those URLs are pushed back into the distributed URL frontier as if never assigned
  3. Assign to healthy workers: The frontier redistributes them to active workers in the next pull cycle

This ensures no URL is lost even when workers disappear mid-crawl.

Graceful Shutdown

Healthy workers can also send explicit "shutting down" signals, allowing the coordinator to immediately reclaim their URLs rather than waiting for heartbeat timeout.

Analogy

It's like a restaurant kitchen: if a chef stops responding to the expeditor's calls (heartbeats), their orders get reassigned to other chefs so customers still get served.

Key Takeaway:
Heartbeat mechanisms detect worker failures, enabling automatic URL reassignment from dead workers to healthy ones, ensuring crawl completeness despite individual machine failures.