Distributed File System Considerations
What you'll learn: How to choose between HDFS and object storage for massive-scale image/video hosting by weighing replication, fault tolerance, and access patterns.
The Core Question
When building a system that stores petabytes of user-uploaded media, you need a distributed file system. Two popular choices emerge: HDFS (Hadoop Distributed File System) and Object Storage (like Amazon S3, Google Cloud Storage). Each excels at different workloads.
HDFS Characteristics
HDFS was designed for big data processing with sequential reads:
- Large block sizes (64–128 MB) optimized for streaming
- Strong consistency with central NameNode coordination
- Built-in 3x replication across DataNodes
- Best for map-reduce jobs processing massive datasets
- Write-once, read-many access pattern
Analogy: HDFS is like a library where you check out entire encyclopedia sets (large blocks) to read cover-to-cover, not for random page lookups.
Object Storage Characteristics
Object storage excels at web-scale random access:
- Optimized for HTTP GET/PUT of individual objects
- Eventual consistency (though many now offer strong consistency)
- Geographic replication across availability zones/regions
- Metadata stored with each object
- Direct CDN integration and pre-signed URLs
- Pay-per-use, elastic scaling
Analogy: Object storage is like a global warehouse where you can instantly retrieve any single item by its tracking number from anywhere.
For Image/Video Hosting: Object Storage Wins
Your access pattern involves:
- Random, individual file retrieval (not batch processing)
- Direct user uploads via pre-signed URLs
- CDN integration for fast delivery
- Geographic distribution for global users
- No need for HDFS's sequential-scan optimization
HDFS would add unnecessary complexity (managing NameNodes, DataNodes) without providing benefits for this workload.
Key Takeaway: For image/video hosting with random access and direct HTTP delivery, object storage provides better fault tolerance, simpler architecture, and natural CDN integration compared to HDFS's block-based design optimized for sequential processing.