Frontier Scalability: Sharding and Distribution
You'll learn: How to distribute a web crawler's URL frontier across multiple machines while preserving per-host politeness guarantees at massive scale.
Why Distribution Matters
A single-machine frontier (even with multi-queue architecture) eventually hits throughput limits. When crawling billions of URLs, you need multiple frontier servers working in parallel. The challenge? Maintaining politeness—ensuring requests to the same host don't scatter across machines, creating accidental DDoS attacks.
Distribution Strategies
Host-Based Sharding
Assign frontier management by hostname hash. All URLs for example.com route to the same frontier server using:
frontier_server = hash(hostname) % num_servers
This guarantees all URLs for a host stay together, so one server enforces politeness per-host.
Analogy: Like assigning ZIP codes to post offices—all mail for one neighborhood goes through the same office that knows delivery schedules.
URL-Based Sharding
Alternatively, shard by full URL hash when host distribution is highly skewed (one domain dominates). The trade-off: you need distributed politeness coordination—multiple servers may handle the same host and must synchronize crawl delays via shared state (e.g., Redis politeness table).
Critical Consideration
Consistent hashing handles server additions/removals gracefully, redistributing only affected hosts rather than reshuffling everything.
Each frontier server maintains its own priority and politeness queues, but now operates on a subset of the hostname space. Crawl workers pull from any available frontier server, respecting global politeness constraints.
Key Takeaway: Sharding the frontier by hostname ensures politeness guarantees scale horizontally, with each server independently managing its assigned hosts' crawl behavior without cross-server coordination overhead.