Deduplication Using Content Hashing
What You'll Learn
You'll learn how to use content hashing to detect duplicate files and dramatically reduce storage costs in image and video hosting systems.
The Problem
When millions of users upload files, many will upload identical content—the same viral image, company logo, or video clip. Storing every copy wastes enormous amounts of storage space and money.
How Content Hashing Works
Before storing an uploaded file, compute its SHA-256 hash—a unique cryptographic fingerprint of the file's contents. Two identical files will always produce the same hash, while even slightly different files produce completely different hashes.
The process:
- User uploads a file
- System computes
SHA-256(file_content)→ e.g.,a3f5b8c2d4e... - Check metadata database: does this hash already exist?
- If yes: Don't store the file again—just create a new metadata record pointing to the existing storage location
- If no: Store the file once and record its hash and location
Storage Architecture
Your metadata database tracks:
file_hash(SHA-256, indexed)storage_location(object storage path)upload_count(reference counter)
Multiple user records can reference the same file_hash, but only one physical copy exists in object storage.
Real-World Impact
If 10,000 users upload the same 5MB image, you store 5MB instead of 50GB—a 99.99% storage savings on duplicates. For platforms hosting billions of files, this translates to millions of dollars saved.
Analogy
Think of a library where identical books are stored once, and borrowers' records point to that single copy rather than buying a new book for each person who wants it.
Key Takeaway: SHA-256 content hashing enables deduplication by creating unique fingerprints of files, allowing you to store each unique file exactly once while serving it to millions of users, reducing storage costs by orders of magnitude.