Course contentsShow
← System Design
Lesson 1841 of 1,91944. Real-World System: Web CrawlerFree lesson

Single-Host Queue Pattern

Learning the pattern of maintaining separate queues per hostname to enforce per-host rate limits and politeness.

Single-Host Queue Pattern

You'll learn how to organize crawl queues by hostname to enforce per-host rate limits and maintain politeness.

The Core Problem

When your web crawler is processing millions of URLs, you can't blast all requests to any single website simultaneously. Doing so violates politeness (overwhelming a server) and often triggers rate limiting or IP bans. You need a way to ensure that requests to the same host are spread out in time.

How Single-Host Queuing Works

The Single-Host Queue Pattern maintains a separate queue for each hostname in your URL frontier. When you discover http://example.com/page1 and http://example.com/page2, both URLs go into the example.com queue—not into one giant shared queue.

The Mechanism

  1. Hash by hostname: Extract the hostname from each URL and route it to a dedicated queue
  2. Sequential processing per host: Each hostname queue is processed by one worker at a time
  3. Enforced delays: Between requests from the same queue, insert a delay (e.g., 1 second)
  4. Independent hosts run in parallel: While waiting for example.com, you can simultaneously crawl another-site.com

Why This Matters

Think of it like airport security lanes—instead of one chaotic line where the same family keeps cutting ahead, each "family" (hostname) gets its own lane. This naturally spaces out their processing while other families move through their lanes independently.

Without this pattern, your priority queue might schedule 100 URLs from example.com back-to-back, violating politeness policies outlined in robots.txt and potentially getting your crawler blocked.

Key Takeaway:
Single-Host Queue Pattern uses separate queues per hostname to enforce per-host politeness delays while maximizing overall crawl parallelism across different domains.