Estimation for Crawler Scale
What you'll learn: How to calculate the resources needed to crawl billions of web pages using back-of-envelope math.
Before building a web crawler, you need to estimate if your design can handle the scale. Let's work through the key metrics using realistic assumptions.
Pages Per Second
Assume you want to crawl 10 billion pages per month and refresh each page every 30 days:
- Pages/day = 10B / 30 ≈ 333 million pages/day
- Pages/second = 333M / 86,400 ≈ 3,850 pages/sec
- With 20% peak buffer: ~4,600 pages/sec
This tells you how many parallel workers you need.
Bandwidth Estimation
Assume average page size is 500 KB (including HTML, CSS, images):
- Bandwidth = 4,600 pages/sec × 500 KB ≈ 2.3 GB/sec or 18.4 Gbps
- For bidirectional traffic and overhead: plan for 25–30 Gbps
This determines your network infrastructure and CDN costs.
Storage Requirements
For 10 billion pages at 500 KB average:
- Raw storage = 10B × 500 KB ≈ 5 PB (petabytes)
- With compression (3:1 ratio): ~1.7 PB
- Add metadata (URLs, timestamps): +10% → ~1.9 PB
Factor in replicas (3x for durability): 5.7 PB total.
Compute Resources
With 200ms average latency per page fetch:
- Concurrent workers needed = 4,600 × 0.2 ≈ 920 workers
- Add parsing/processing overhead (2x): ~2,000 compute cores
- CPU requirements: moderate (I/O-bound workload)
- Memory: ~2 GB per worker → 4 TB RAM across cluster
Key Takeaway: Back-of-envelope math converts business requirements (pages/month) into concrete infrastructure needs (servers, bandwidth, storage), helping you validate feasibility and estimate costs before implementation.