Coordinator-Worker Pattern for Crawling
What You'll Learn
You'll learn how a central coordinator distributes crawling work across multiple worker nodes to achieve massive parallelism while maintaining control and coordination.
The Core Idea
The Coordinator-Worker Pattern splits your web crawler into two distinct roles:
- Coordinator: A central brain that manages the URL frontier, assigns batches of URLs to workers, tracks progress, and handles failures.
- Workers: Multiple independent machines that receive URL assignments, perform HTTP requests, parse content, extract links, and report results back to the coordinator.
Think of it like a restaurant: the coordinator is the expediter who receives orders and assigns them to chefs (workers). Each chef works independently on their assigned dishes, but the expediter ensures no two chefs cook the same dish and that orders are distributed fairly.
How It Works
The coordinator maintains the URL frontier and periodically:
- Pulls batches of URLs (e.g., 100-1000 URLs) from the frontier
- Checks which workers are available
- Assigns URL batches to workers via RPC or message queues
- Tracks which batches are in-progress vs completed
Workers independently:
- Receive their URL batch
- Crawl each URL (respecting politeness, robots.txt)
- Extract new URLs and content
- Send discovered URLs and crawl status back to coordinator
- Request the next batch
The coordinator handles worker failures by reassigning stale batches (using timeouts or heartbeats) and prevents duplicate work through centralized URL deduplication.
Why This Pattern?
This design cleanly separates control plane (coordinator) from data plane (workers), allowing you to scale workers horizontally while keeping coordination logic centralized and simpler.
Key Takeaway: The Coordinator-Worker pattern enables distributed web crawling by having a central coordinator manage the URL frontier and assign work batches to stateless worker nodes, achieving both parallelism and coordination without complex distributed consensus.