Course contentsShow
← System Design
Lesson 1868 of 1,91944. Real-World System: Web CrawlerFree lesson

Coordinating Politeness Across Workers

Ensuring rate limits and delays are respected when multiple workers crawl the same domain simultaneously.

Coordinating Politeness Across Workers

You'll learn how to prevent multiple crawler workers from overwhelming a domain when they independently crawl URLs from the same host.

The Challenge

In a distributed crawler, you've implemented politeness rules on each worker—respecting crawl delays and robots.txt. But here's the problem: if Worker A and Worker B both receive URLs from example.com, they might both crawl it simultaneously, effectively doubling your request rate and violating politeness guarantees.

Analogy: Imagine two cashiers at separate registers, both calling the same customer at once. Even if each cashier waits politely between calls, together they're harassing the customer.

Coordination Strategies

Host-Based Sharding: Assign each host to a specific worker using consistent hashing. When distributing URLs from the coordinator's frontier, hash the hostname and route all example.com URLs to Worker 3, all news.org URLs to Worker 7, etc. This guarantees only one worker ever touches a given domain.

Centralized Politeness Service: Workers consult a shared politeness table (Redis or similar) before crawling. They acquire a "lease" for a host, ensuring exclusive access for a time window. Other workers must wait or skip URLs for that host.

Distributed Politeness Table: Each worker maintains local politeness state but synchronizes with peers. Before crawling example.com, Worker A checks: "Is anyone else crawling this?" Uses distributed locks or a coordination service like ZooKeeper.

Trade-offs

Host sharding is simple but creates hot workers (popular domains → uneven load). Centralized services add latency but offer precise control. Choose based on your scale and politeness strictness requirements.

Key Takeaway: Without coordination, distributed workers can accidentally violate politeness rules; use host-based sharding or shared state to ensure only one worker crawls each domain at a time.