Object Size Limits and Multipart Uploads
What you'll learn: How to handle large files in object storage and when multipart uploads become necessary.
Why Object Size Matters
Cloud object storage services impose maximum size limits on individual objects. For example, AWS S3 allows a single PUT operation to upload files up to 5GB, but the absolute maximum object size is 5TB. How do you bridge this gap? That's where multipart uploads come in.
Understanding Multipart Uploads
Think of multipart upload like mailing a large poster: instead of folding it into one envelope (risking damage), you cut it into smaller sections, mail each piece separately, and reassemble them at the destination.
When uploading files larger than 100MB, you should use multipart upload. Here's why:
- Reliability: If one part fails, you only retry that piece, not the entire file
- Speed: Multiple parts upload in parallel, utilizing your bandwidth better
- Resumability: You can pause and resume large uploads without starting over
- Efficiency: Network hiccups don't force you to re-upload gigabytes of data
How It Works
The process breaks your large file into chunks (typically 5MB to 5GB each), uploads each part independently, then tells the storage service to stitch them back together. The service ensures all parts arrived and creates the complete object.
For files under 100MB, a simple single PUT request works fine—multipart adds unnecessary complexity. But once you cross that threshold or approach the 5GB single-operation limit, multipart becomes essential.
Key Takeaway: Object storage has size limits per operation (often 5GB for single uploads) and per object (often 5TB total). Use multipart uploads for files over 100MB to gain reliability, speed, and the ability to upload objects up to the maximum size limit.