DNS Prefetching and Batch Resolution
What you'll learn: How to dramatically boost crawler throughput by resolving DNS records ahead of time and in batches rather than one-by-one.
When your crawler operates at massive scale, waiting for DNS resolution on every single URL becomes a severe bottleneck. Each lookup can take 20-100ms, and if you're crawling millions of pages, those milliseconds compound into hours of wasted time.
The Core Problem
Traditional crawlers resolve DNS serially: fetch URL from frontier → resolve DNS → make HTTP request → repeat. This means your crawler threads spend most of their time idle, waiting for DNS responses.
DNS Prefetching Strategy
Prefetching means resolving DNS records before you actually need them. As soon as URLs enter your frontier, you extract their hostnames and send DNS queries asynchronously. By the time a worker thread picks up that URL for crawling, the IP address is already cached and ready.
Think of it like a restaurant kitchen: instead of waiting for a customer to order before you start cooking, you prep ingredients in advance based on what's queued up.
Batch Resolution
Rather than sending one DNS query at a time, you batch multiple hostnames together and resolve them in parallel. Modern DNS libraries support asynchronous batch queries, letting you send hundreds of lookups simultaneously and process responses as they arrive.
For example, when your frontier loads 1,000 new URLs spanning 200 unique domains, immediately fire off all 200 DNS queries at once rather than sequentially.
Implementation Points
- Run a dedicated DNS resolution worker pool separate from your HTTP fetchers
- Maintain a "DNS pending" queue where hostnames wait for resolution
- Use asynchronous DNS libraries (e.g.,
aiodns,dnspythonwith async support) - Coordinate with your DNS cache (from lesson 1857) to avoid redundant lookups
Key Takeaway: DNS prefetching and batch resolution transform DNS from a per-request blocking operation into an asynchronous pipeline stage, keeping your crawler threads productive and dramatically increasing throughput.