Robots.txt and Crawl Etiquette
What you'll learn: How web crawlers respect site-level directives and follow ethical crawling practices to avoid overwhelming servers.
What Is Robots.txt?
When you build a web crawler, you can't just scrape anything you want—websites have rules. The robots.txt file is a standard protocol that tells crawlers which parts of a site they're allowed to visit. Think of it as a "No Trespassing" sign on certain digital paths.
This file lives at the root of a domain (e.g., https://example.com/robots.txt) and uses simple directives:
User-agent: *
Disallow: /admin/
Disallow: /private/
Crawl-delay: 10
This says: "All crawlers must avoid /admin/ and /private/ directories, and wait 10 seconds between requests."
Why Crawl Etiquette Matters
Beyond robots.txt, responsible crawlers follow ethical practices:
- Respect crawl-delay: Don't hammer servers with rapid-fire requests. If
robots.txtspecifies a delay, honor it. If not, use sensible defaults (1-2 seconds). - Identify yourself: Set a proper User-Agent header so site owners know who's crawling and can contact you if needed.
- Handle rate limits gracefully: Back off when you receive 429 (Too Many Requests) or 503 (Service Unavailable) responses.
- Check robots.txt regularly: Sites update their rules; cache the file but refresh it periodically (every 24 hours is common).
Implementation Approach
Before crawling a domain, fetch and parse its robots.txt. Store allowed/disallowed patterns per domain in your URL frontier. Before dequeuing a URL, verify it's permitted and check if sufficient time has passed since your last request to that domain.
Key Takeaway: Respecting robots.txt and implementing crawl delays isn't just polite—it's essential for being a good internet citizen and avoiding legal issues or IP bans.