HTTP Request Handling
What you'll learn: How a web crawler's fetcher component manages HTTP requests, responses, and edge cases to reliably retrieve web pages.
The Core Challenge
When your crawler visits millions of URLs, it must speak HTTP fluently—sending proper requests, understanding what servers say back, and handling the messy reality of the web. The fetcher component acts as the crawler's diplomat, negotiating with web servers using the right protocol, patience, and grace under failure.
HTTP Methods and Headers
Use GET for fetching pages (not POST, which modifies data). Always set a descriptive User-Agent header identifying your crawler—it's basic etiquette and helps webmasters contact you if problems arise. Include Accept-Encoding: gzip to reduce bandwidth, and respect If-Modified-Since for politeness caching.
Status Codes: The Server's Reply
- 2xx (Success): Download and process the content
- 3xx (Redirect): Follow
Locationheader, but limit redirect chains (5-7 max) to avoid loops - 4xx (Client Error): 404 means dead link—don't retry. 429 means "slow down"—respect it
- 5xx (Server Error): Temporary issues—retry with exponential backoff
Redirects and Chains
Follow redirects automatically but track the chain. If you hit 10 redirects, something's wrong—bail out. Update your URL frontier with the final canonical URL to avoid re-crawling the same content through different redirect paths.
Timeouts: Don't Wait Forever
Set connection timeouts (e.g., 5 seconds) and read timeouts (e.g., 30 seconds). Some servers are slow or malicious—your crawler can't afford to hang indefinitely. Failed requests go back to the queue for later retry, not into an eternal wait.
Key Takeaway: Robust HTTP handling means using proper methods and headers, interpreting status codes intelligently, limiting redirect chains, and enforcing timeouts to keep your crawler efficient and respectful.