Course contentsShow
← System Design
Lesson 1840 of 1,91944. Real-World System: Web CrawlerFree lesson

Politeness Requirements for Web Crawling

Understanding politeness constraints: rate limiting per host, respecting robots.txt, and avoiding server overload.

Politeness Requirements for Web Crawling

What you'll learn: How to prevent your crawler from overwhelming or disrupting the websites it visits through rate limiting, protocol compliance, and resource-aware scheduling.


The Courtesy Problem

A web crawler can issue thousands of requests per second. Without constraints, even a well-intentioned crawler becomes a Distributed Denial-of-Service (DDoS) attack against target servers. Politeness requirements ensure your crawler behaves like a respectful visitor, not a resource hog.


Core Politeness Constraints

Rate Limiting Per Host
Never bombard a single server with rapid-fire requests. Implement per-host delays (e.g., 1–5 seconds between requests to example.com). This prevents:

  • Overwhelming server CPU/bandwidth
  • Triggering anti-bot defenses
  • Degrading service for real users

Think of it like knocking on someone's door: knock once, wait for an answer, then knock again—don't pound continuously.

Respecting robots.txt
As covered earlier, always parse and honor robots.txt directives:

  • Disallow paths you must skip
  • Crawl-delay specifies minimum wait time between requests
  • Ignoring these rules violates web etiquette and may lead to IP bans

Avoiding Server Overload
Beyond per-host delays:

  • Respect HTTP 429 (Too Many Requests) responses—back off exponentially
  • Limit concurrent connections per domain (e.g., max 2–3 simultaneous sockets)
  • Crawl during off-peak hours when possible for high-traffic sites

Implementation Pattern

Your URL Frontier must enforce politeness:

  • Group URLs by hostname
  • Track last-crawled timestamp per host
  • Only dequeue a URL if sufficient time has elapsed since the last request to that host

This ensures fairness: even if your frontier has 1,000 URLs from siteA.com, you process them slowly, interspersed with URLs from other domains.


Key Takeaway:
Politeness constraints—per-host rate limiting, robots.txt compliance, and server-aware scheduling—transform your crawler from a potential threat into a responsible web citizen.