Course contentsShow
← System Design
Lesson 1832 of 1,91944. Real-World System: Web CrawlerFree lesson

HTTP Request Handling

Managing HTTP methods, headers, status codes, redirects, and timeouts in the fetcher component.

HTTP Request Handling

What you'll learn: How a web crawler's fetcher component manages HTTP requests, responses, and edge cases to reliably retrieve web pages.

The Core Challenge

When your crawler visits millions of URLs, it must speak HTTP fluently—sending proper requests, understanding what servers say back, and handling the messy reality of the web. The fetcher component acts as the crawler's diplomat, negotiating with web servers using the right protocol, patience, and grace under failure.

HTTP Methods and Headers

Use GET for fetching pages (not POST, which modifies data). Always set a descriptive User-Agent header identifying your crawler—it's basic etiquette and helps webmasters contact you if problems arise. Include Accept-Encoding: gzip to reduce bandwidth, and respect If-Modified-Since for politeness caching.

Status Codes: The Server's Reply

  • 2xx (Success): Download and process the content
  • 3xx (Redirect): Follow Location header, but limit redirect chains (5-7 max) to avoid loops
  • 4xx (Client Error): 404 means dead link—don't retry. 429 means "slow down"—respect it
  • 5xx (Server Error): Temporary issues—retry with exponential backoff

Redirects and Chains

Follow redirects automatically but track the chain. If you hit 10 redirects, something's wrong—bail out. Update your URL frontier with the final canonical URL to avoid re-crawling the same content through different redirect paths.

Timeouts: Don't Wait Forever

Set connection timeouts (e.g., 5 seconds) and read timeouts (e.g., 30 seconds). Some servers are slow or malicious—your crawler can't afford to hang indefinitely. Failed requests go back to the queue for later retry, not into an eternal wait.

Key Takeaway: Robust HTTP handling means using proper methods and headers, interpreting status codes intelligently, limiting redirect chains, and enforcing timeouts to keep your crawler efficient and respectful.