Worker Health Monitoring and Failover
What You'll Learn
You'll learn how to detect when crawler workers crash or stall and automatically reassign their in-progress URLs to healthy workers.
The Problem
In a distributed web crawler with many workers, machines fail, processes crash, or workers hang. Without monitoring, URLs assigned to dead workers would never get crawled, creating black holes in your crawl coverage.
Core Mechanism: Heartbeats
Each worker sends periodic heartbeat signals to the coordinator—simple "I'm alive" messages every 10-30 seconds. Think of it like a pulse oximeter: if the beeps stop, something's wrong.
The coordinator tracks the last heartbeat timestamp for each worker. If a worker misses multiple consecutive heartbeats (typically 2-3 cycles), the coordinator marks it as unhealthy or dead.
Reassignment Strategy
When a worker is declared failed:
- Retrieve pending URLs: The coordinator looks up which URLs were assigned to that worker (often stored in a "worker_id → URL_list" mapping in Redis or a database)
- Return to frontier: Those URLs are pushed back into the distributed URL frontier as if never assigned
- Assign to healthy workers: The frontier redistributes them to active workers in the next pull cycle
This ensures no URL is lost even when workers disappear mid-crawl.
Graceful Shutdown
Healthy workers can also send explicit "shutting down" signals, allowing the coordinator to immediately reclaim their URLs rather than waiting for heartbeat timeout.
Analogy
It's like a restaurant kitchen: if a chef stops responding to the expeditor's calls (heartbeats), their orders get reassigned to other chefs so customers still get served.
Key Takeaway:
Heartbeat mechanisms detect worker failures, enabling automatic URL reassignment from dead workers to healthy ones, ensuring crawl completeness despite individual machine failures.