Politeness Table and Per-Host State
You'll learn how to track per-host crawling state to enforce politeness constraints across a distributed web crawler.
What It Is
A politeness table is an in-memory data structure that maintains crawling metadata for each host your crawler interacts with. It stores critical timing and state information that ensures you respect crawl delays, avoid overwhelming servers, and maintain fair resource allocation across hosts.
Why Per-Host State Matters
When you extract a URL from a back queue (which contains only URLs from a single host), you need to decide:
- Can I fetch from this host right now?
- When was the last fetch to this host?
- How many active connections do I have to this host?
Without tracking this state, you'd either crawl too aggressively (violating politeness) or too conservatively (wasting resources).
Core State Tracked
For each host, the politeness table maintains:
- Last fetch time: Timestamp of the most recent successful request
- Next allowed fetch time: Computed as
last_fetch_time + crawl_delay, prevents fetching too soon - Active connection count: Number of in-flight HTTP requests to prevent overwhelming the host with concurrent connections
Think of it like a traffic light system for each host—green means "ready to fetch," red means "wait," and the connection counter ensures you don't send a convoy when one car would do.
Usage Pattern
Before issuing an HTTP request, the crawler checks the politeness table: if current time < next allowed fetch time OR active connections >= limit, skip this host and try another. After fetching, update last fetch time and decrement active connections when done.
Key Takeaway: The politeness table is your crawler's memory—it remembers when each host was last visited and how busy it currently is, ensuring polite, efficient crawling at scale.