Course contentsShow
← System Design
Lesson 1836 of 1,91944. Real-World System: Web CrawlerFree lesson

Estimation for Crawler Scale

Back-of-envelope calculations for pages per second, bandwidth, storage, and compute resources at web scale.

Estimation for Crawler Scale

What you'll learn: How to calculate the resources needed to crawl billions of web pages using back-of-envelope math.

Before building a web crawler, you need to estimate if your design can handle the scale. Let's work through the key metrics using realistic assumptions.

Pages Per Second

Assume you want to crawl 10 billion pages per month and refresh each page every 30 days:

  • Pages/day = 10B / 30 ≈ 333 million pages/day
  • Pages/second = 333M / 86,400 ≈ 3,850 pages/sec
  • With 20% peak buffer: ~4,600 pages/sec

This tells you how many parallel workers you need.

Bandwidth Estimation

Assume average page size is 500 KB (including HTML, CSS, images):

  • Bandwidth = 4,600 pages/sec × 500 KB ≈ 2.3 GB/sec or 18.4 Gbps
  • For bidirectional traffic and overhead: plan for 25–30 Gbps

This determines your network infrastructure and CDN costs.

Storage Requirements

For 10 billion pages at 500 KB average:

  • Raw storage = 10B × 500 KB ≈ 5 PB (petabytes)
  • With compression (3:1 ratio): ~1.7 PB
  • Add metadata (URLs, timestamps): +10% → ~1.9 PB

Factor in replicas (3x for durability): 5.7 PB total.

Compute Resources

With 200ms average latency per page fetch:

  • Concurrent workers needed = 4,600 × 0.2 ≈ 920 workers
  • Add parsing/processing overhead (2x): ~2,000 compute cores
  • CPU requirements: moderate (I/O-bound workload)
  • Memory: ~2 GB per worker → 4 TB RAM across cluster

Key Takeaway: Back-of-envelope math converts business requirements (pages/month) into concrete infrastructure needs (servers, bandwidth, storage), helping you validate feasibility and estimate costs before implementation.