Queue Router and Host Mapping
What you'll learn: How a web crawler's queue router intelligently distributes URLs to specific back queues based on their hostname to enforce politeness.
The Routing Challenge
You have multiple front queues managing priority and multiple back queues enforcing per-host politeness. But how does a URL get from the front to the correct back queue? You need a routing layer that maps URLs to the right destination based on their hostname.
How Queue Routing Works
The queue router sits between front and back queues. When a URL is dequeued from a front queue, the router:
- Extracts the hostname (e.g.,
www.example.comfromhttps://www.example.com/page) - Maps the hostname to a back queue using a deterministic function
- Enqueues the URL to that specific back queue
This ensures all URLs from the same host go to the same back queue, where politeness delays are enforced.
Mapping Strategy
The simplest approach is hash-based mapping:
back_queue_id = hash(hostname) % number_of_back_queues
This distributes hosts evenly across back queues while guaranteeing the same host always maps to the same queue. The router maintains a hostname-to-queue mapping table for quick lookups.
Why This Matters
Without proper routing, URLs from example.com could scatter across multiple back queues, breaking politeness enforcement. The router acts like a traffic cop, directing each host's URLs to their dedicated lane where crawl delays are respected.
Key Takeaway:
The queue router uses hostname-based mapping to deterministically route URLs from front queues to specific back queues, ensuring all requests to the same host flow through a single politeness-controlled channel.