Course contentsShow
← System Design
Lesson 1837 of 1,91944. Real-World System: Web CrawlerFree lesson

Single-Machine Crawler Limitations

Bottlenecks in CPU, network, and storage that necessitate distributed crawler architecture for scale.

Single-Machine Crawler Limitations

What you'll learn: Why a single machine cannot handle web-scale crawling and what specific bottlenecks force us toward distributed architecture.

When you estimate crawler scale (billions of pages, petabytes of data, continuous re-crawls), a single machine quickly hits hard limits. Think of it like trying to vacuum every street in a country with one vacuum cleaner—no matter how good the vacuum, you'll run into fundamental physical constraints.

CPU Bottleneck

HTML parsing, URL extraction, content deduplication hashing, and robots.txt processing are CPU-intensive. Parsing millions of pages per day saturates even multi-core processors. Your crawler spends significant cycles tokenizing HTML, normalizing URLs, and detecting duplicate content—work that grows linearly with page count.

Network Bottleneck

A single network interface has finite bandwidth (typically 1-10 Gbps). When crawling, you're both downloading pages (inbound) and making HTTP requests (outbound). At scale, you'll saturate the NIC before exhausting CPU or memory. Even with compression, fetching billions of pages requires aggregate bandwidth no single machine possesses.

Storage Bottleneck

Storing raw HTML, extracted metadata, and crawl state (visited URLs, queue) requires vast disk space. A single disk's I/O throughput (IOPS) becomes the limiting factor when persisting thousands of pages per second. SSDs help but don't eliminate the fundamental constraint—you need more physical drives.

Memory Constraints

Keeping visited URL hashes, bloom filters for deduplication, and the URL frontier (crawl queue) in memory is critical for performance. With billions of URLs, even space-efficient data structures exceed typical RAM limits (64-512 GB).

Key Takeaway: Single-machine crawlers fail at web scale due to CPU saturation during parsing, network bandwidth limits, disk I/O constraints, and insufficient memory for state management—necessitating horizontal distribution across many machines.