Course contentsShow
← System Design
Lesson 1852 of 1,91944. Real-World System: Web CrawlerFree lesson

Content Fingerprinting with Hashing

Use cryptographic hashes like SHA-256 to create fingerprints of page content for detecting duplicate pages.

Content Fingerprinting with Hashing

What you'll learn: How web crawlers use cryptographic hashes to efficiently detect and skip duplicate pages at massive scale.

The Problem: Finding Duplicates in Billions of Pages

When a web crawler fetches pages, it encounters the same content under different URLs constantly—mirrors, copied content, scraped pages, or identical documents hosted in multiple locations. Storing or indexing the same content multiple times wastes storage, compute, and degrades search quality.

How Content Fingerprinting Works

Instead of comparing entire page contents byte-by-byte (impossibly slow at scale), we create a fingerprint: a fixed-size hash that uniquely represents the content.

The Process:

  1. Extract content from the fetched page (after HTML parsing)
  2. Apply a cryptographic hash function like SHA-256 to the content
  3. Store the hash (32 bytes for SHA-256) in a seen-content set
  4. Before indexing, check if this hash already exists
  5. If duplicate, skip processing; if new, index and store the fingerprint

Why SHA-256?

Cryptographic hashes like SHA-256 offer:

  • Deterministic: Same content always produces the same hash
  • Fixed size: Any content → exactly 256 bits (32 bytes)
  • Collision-resistant: Virtually impossible for two different pages to produce the same hash
  • Fast: Can process megabytes of content in milliseconds

Analogy: Think of it like taking a photograph's unique serial number instead of storing thousands of copies of the same photo.

Storage Efficiency

Comparing a 32-byte hash is far cheaper than comparing multi-kilobyte HTML documents. A Bloom filter or distributed hash table of fingerprints lets you detect duplicates with minimal memory.

Key Takeaway: Content fingerprinting with SHA-256 transforms the duplicate detection problem from comparing massive text documents to simple 32-byte hash lookups, enabling web crawlers to efficiently skip billions of duplicate pages.