Course contentsShow
← System Design
Lesson 1842 of 1,91944. Real-World System: Web CrawlerFree lesson

Politeness Budget and Crawl Delay

Implementing configurable delays between requests to the same host based on robots.txt directives or policies.

Politeness Budget and Crawl Delay

What you'll learn: How to implement configurable delays between requests to respect server resources and avoid overwhelming target hosts.

The Core Concept

A politeness budget is the allowance of resources (requests, bandwidth, time) your crawler may consume from a single host within a given time window. Crawl delay is the enforced wait time between consecutive requests to the same domain, ensuring you don't hammer servers with rapid-fire requests.

Think of it like visiting a buffet: you could pile your plate high every trip, but restaurant etiquette (and house rules) say you should wait between visits and not monopolize the food. Similarly, even though your crawler can fetch 100 pages per second, you should respect the host's capacity.

How It Works

  1. Read robots.txt directives: Many sites specify a Crawl-delay: N value (in seconds) telling crawlers to wait N seconds between requests
  2. Configurable policies: Your crawler maintains per-host settings—either from robots.txt, default policies (e.g., "1 request every 2 seconds"), or custom overrides for high-priority domains
  3. Enforcement in the frontier: Before dispatching a URL from a single-host queue, check the timestamp of the last request to that host. If insufficient time has passed, delay or re-queue the URL

Why It Matters

Without crawl delays, your distributed crawler might send thousands of simultaneous requests to a small blog, effectively DDoS-ing it. Politeness budgets prevent you from being blocked, maintain good relationships with site operators, and ensure sustainable, long-term crawling access.

Key Takeaway:
Politeness budget and crawl delay transform your crawler from a resource hog into a respectful citizen of the web, enforcing configurable wait times between requests to the same host based on robots.txt or internal policies.