Course contentsShow
← System Design
Lesson 1594 of 1,91939. Real-World System: Image and Video HostingFree lesson

Storage Sharding by User or Time

Partition storage by user ID or upload timestamp to distribute load and enable efficient data lifecycle management.

Storage Sharding by User or Time

What you'll learn: How to partition image and video storage across multiple physical systems based on user ID or upload timestamp to balance load and simplify data lifecycle management.

The Core Problem

As your image/video hosting platform grows to billions of files, a single storage cluster becomes a bottleneck. You need a systematic way to distribute files across multiple storage systems while maintaining fast lookups and efficient management.

Sharding by User ID

Partition storage based on the uploader's user ID using a consistent hash or modulo operation:

storage_shard = user_id % number_of_shards

Benefits:

  • All of a user's content lives together, simplifying queries like "show all user uploads"
  • User-level operations (delete account, enforce quotas) are efficient
  • Natural isolation for privacy and access control

Tradeoffs:

  • Popular users (celebrities, brands) create hot shards
  • Growth is uneven across shards

Sharding by Upload Time

Partition storage based on upload timestamp, creating time-based buckets (e.g., monthly shards):

storage_shard = "uploads-2024-03"

Benefits:

  • Predictable, even distribution of writes over time
  • Simplifies data lifecycle: older shards become read-only, can be archived to cold storage
  • Easy to implement retention policies (delete all data from 2020)
  • Recent uploads (which see most traffic) are isolated from old data

Tradeoffs:

  • User-centric queries span multiple shards
  • Requires time-based routing logic

The Analogy

Think of a filing cabinet: User-based sharding is like organizing by customer name (all documents per customer together), while time-based sharding is like organizing by year (easy to archive old years, but one customer's files are scattered).

Key Takeaway: Shard storage by user ID for user-centric operations or by upload time for predictable growth and simplified archival—hybrid approaches combining both dimensions are common in production systems.