Course contentsShow
← System Design
Lesson 1831 of 1,91944. Real-World System: Web CrawlerFree lesson

Robots.txt and Crawl Etiquette

Understanding robots.txt protocol, respecting site-level crawl directives, and ethical crawling practices.

Robots.txt and Crawl Etiquette

What you'll learn: How web crawlers respect site-level directives and follow ethical crawling practices to avoid overwhelming servers.

What Is Robots.txt?

When you build a web crawler, you can't just scrape anything you want—websites have rules. The robots.txt file is a standard protocol that tells crawlers which parts of a site they're allowed to visit. Think of it as a "No Trespassing" sign on certain digital paths.

This file lives at the root of a domain (e.g., https://example.com/robots.txt) and uses simple directives:

User-agent: *
Disallow: /admin/
Disallow: /private/
Crawl-delay: 10

This says: "All crawlers must avoid /admin/ and /private/ directories, and wait 10 seconds between requests."

Why Crawl Etiquette Matters

Beyond robots.txt, responsible crawlers follow ethical practices:

  • Respect crawl-delay: Don't hammer servers with rapid-fire requests. If robots.txt specifies a delay, honor it. If not, use sensible defaults (1-2 seconds).
  • Identify yourself: Set a proper User-Agent header so site owners know who's crawling and can contact you if needed.
  • Handle rate limits gracefully: Back off when you receive 429 (Too Many Requests) or 503 (Service Unavailable) responses.
  • Check robots.txt regularly: Sites update their rules; cache the file but refresh it periodically (every 24 hours is common).

Implementation Approach

Before crawling a domain, fetch and parse its robots.txt. Store allowed/disallowed patterns per domain in your URL frontier. Before dequeuing a URL, verify it's permitted and check if sufficient time has passed since your last request to that domain.

Key Takeaway: Respecting robots.txt and implementing crawl delays isn't just polite—it's essential for being a good internet citizen and avoiding legal issues or IP bans.