Course contentsShow
← System Design
Lesson 1870 of 1,91944. Real-World System: Web CrawlerFree lesson

Content Storage and Deduplication

Designing distributed storage for crawled content with hashing to detect and eliminate duplicate pages.

Content Storage and Deduplication

What you'll learn: How to design distributed storage for billions of crawled web pages while efficiently detecting and eliminating duplicates.

The Challenge

A large-scale web crawler fetches billions of pages daily. Storing every version wastes enormous space—many pages are duplicates (mirrors, scraper sites, unchanged recrawls). You need a strategy to store content efficiently and avoid processing the same content twice.

Core Design Strategy

Content-Addressable Storage

Instead of storing content by arbitrary IDs, use content hashing (SHA-256 or similar) to generate a unique fingerprint for each page. The hash becomes the storage key:

hash = SHA256(page_content)
storage_key = hash

If two pages produce the same hash, they're identical—store only one copy. Think of it like a library where books are shelved by their ISBN: if two "different" books have the same ISBN, they're actually the same book.

Storage Architecture

  • Blob Store: Use distributed object storage (S3, HDFS) to hold actual HTML/content, keyed by hash
  • Metadata Database: Store URL → hash mappings, crawl timestamps, and headers separately
  • Deduplication Layer: Before saving, compute hash and check if it exists; if yes, just add new URL → hash reference

Near-Duplicate Handling

Recall Simhash from lesson 1855: for near-duplicates (pages differing by ads/timestamps), use locality-sensitive hashing to cluster similar content and store only canonical versions.

Distributed Coordination

Workers compute hashes locally, then query a shared deduplication service (backed by your distributed Bloom filter or hash index) before uploading to blob storage.

Key Takeaway:
Content-addressable storage with cryptographic hashing transforms duplicate detection into a simple key-lookup problem, saving massive storage while enabling efficient distributed writes.