Course contentsShow
← System Design
Lesson 1862 of 1,91944. Real-World System: Web CrawlerFree lesson

Why Distribute a Web Crawler

Understanding the limitations of single-machine crawlers and the need for distributed architecture at scale.

Why Distribute a Web Crawler

You'll learn: Why a single-machine web crawler cannot efficiently index the modern web at scale.

The Single-Machine Bottleneck

After exploring robots.txt caching and parsing, we've built a solid foundation for a single-machine crawler. But here's the reality: one machine simply cannot crawl the entire internet.

Consider the numbers: the web contains tens of billions of pages, and new content appears every second. A single machine faces hard limits:

  • Bandwidth ceiling: Even a high-speed connection (1 Gbps) can only download so much per second. Fetching billions of pages at reasonable speeds is physically impossible from one location.
  • CPU constraints: Parsing HTML, extracting links, and computing content fingerprints are CPU-intensive. One processor can't keep up.
  • Memory limits: The URL Frontier, Bloom filters for deduplication, DNS cache, and robots.txt cache all compete for RAM. A single machine runs out quickly.
  • Disk I/O saturation: Writing crawled content and persisting frontier state creates I/O bottlenecks.
  • Geographic latency: Crawling sites hosted worldwide from a single data center means high latencies and slow throughput.

The Need for Distribution

Think of a single crawler like one person trying to read every book in every library on Earth. It's not just slow—it's impossible within any reasonable timeframe.

Distributed crawling solves this by:

  • Splitting the workload across hundreds or thousands of machines
  • Placing crawlers geographically closer to target websites
  • Parallelizing DNS resolution, content fetching, and parsing
  • Scaling storage and memory horizontally

Key Takeaway: Single-machine crawlers hit fundamental physical limits (bandwidth, CPU, memory, I/O) that make web-scale crawling impossible; distribution is not an optimization—it's a necessity for indexing billions of pages efficiently.