Course contentsShow
← System Design
Lesson 1848 of 1,91944. Real-World System: Web CrawlerFree lesson

Politeness Table and Per-Host State

Maintaining state for each host including last fetch time, next allowed fetch time, and active connection counts.

Politeness Table and Per-Host State

You'll learn how to track per-host crawling state to enforce politeness constraints across a distributed web crawler.

What It Is

A politeness table is an in-memory data structure that maintains crawling metadata for each host your crawler interacts with. It stores critical timing and state information that ensures you respect crawl delays, avoid overwhelming servers, and maintain fair resource allocation across hosts.

Why Per-Host State Matters

When you extract a URL from a back queue (which contains only URLs from a single host), you need to decide:

  • Can I fetch from this host right now?
  • When was the last fetch to this host?
  • How many active connections do I have to this host?

Without tracking this state, you'd either crawl too aggressively (violating politeness) or too conservatively (wasting resources).

Core State Tracked

For each host, the politeness table maintains:

  • Last fetch time: Timestamp of the most recent successful request
  • Next allowed fetch time: Computed as last_fetch_time + crawl_delay, prevents fetching too soon
  • Active connection count: Number of in-flight HTTP requests to prevent overwhelming the host with concurrent connections

Think of it like a traffic light system for each host—green means "ready to fetch," red means "wait," and the connection counter ensures you don't send a convoy when one car would do.

Usage Pattern

Before issuing an HTTP request, the crawler checks the politeness table: if current time < next allowed fetch time OR active connections >= limit, skip this host and try another. After fetching, update last fetch time and decrement active connections when done.

Key Takeaway: The politeness table is your crawler's memory—it remembers when each host was last visited and how busy it currently is, ensuring polite, efficient crawling at scale.