Content Storage and Deduplication
What you'll learn: How to design distributed storage for billions of crawled web pages while efficiently detecting and eliminating duplicates.
The Challenge
A large-scale web crawler fetches billions of pages daily. Storing every version wastes enormous space—many pages are duplicates (mirrors, scraper sites, unchanged recrawls). You need a strategy to store content efficiently and avoid processing the same content twice.
Core Design Strategy
Content-Addressable Storage
Instead of storing content by arbitrary IDs, use content hashing (SHA-256 or similar) to generate a unique fingerprint for each page. The hash becomes the storage key:
hash = SHA256(page_content)
storage_key = hash
If two pages produce the same hash, they're identical—store only one copy. Think of it like a library where books are shelved by their ISBN: if two "different" books have the same ISBN, they're actually the same book.
Storage Architecture
- Blob Store: Use distributed object storage (S3, HDFS) to hold actual HTML/content, keyed by hash
- Metadata Database: Store URL → hash mappings, crawl timestamps, and headers separately
- Deduplication Layer: Before saving, compute hash and check if it exists; if yes, just add new URL → hash reference
Near-Duplicate Handling
Recall Simhash from lesson 1855: for near-duplicates (pages differing by ads/timestamps), use locality-sensitive hashing to cluster similar content and store only canonical versions.
Distributed Coordination
Workers compute hashes locally, then query a shared deduplication service (backed by your distributed Bloom filter or hash index) before uploading to blob storage.
Key Takeaway:
Content-addressable storage with cryptographic hashing transforms duplicate detection into a simple key-lookup problem, saving massive storage while enabling efficient distributed writes.