Back Queue: Politeness Enforcement
What you'll learn: How the back queue layer ensures your crawler respects per-host rate limits and avoids hammering servers.
The Core Idea
While the front queue decides which URLs are important to crawl, the back queue enforces when you can actually fetch them, based on politeness rules. This layer maintains one queue per host and tracks the last access time for each host to enforce crawl delays.
Think of it like a traffic light system at a busy intersection: the front queue decides which cars (URLs) get priority to enter the intersection, but the back queue controls the exact timing of when each car can go through, ensuring no lane (host) gets overwhelmed.
How It Works
When a URL moves from the front queue to the back queue:
- Host extraction: Parse the URL to identify its hostname (e.g.,
example.com) - Queue assignment: Route the URL to the dedicated queue for that host
- Delay enforcement: Before dequeuing for fetch, check if enough time has passed since the last request to that host
- Blocking: If the crawl delay hasn't elapsed, the queue remains blocked until the timer expires
For example, if robots.txt specifies a 2-second crawl delay for news.com, the back queue for news.com will only release one URL every 2 seconds, regardless of how many URLs are waiting.
Front Queue → Prioritizer → [Back Queue: news.com] → Wait 2s → Fetcher
→ [Back Queue: blog.org] → Wait 1s → Fetcher
→ [Back Queue: shop.io ] → Wait 5s → Fetcher
Key Takeaway: The back queue layer implements per-host queues that enforce politeness by blocking URL dequeuing until the required crawl delay has passed, protecting destination servers from being overwhelmed.