Hot vs Cold Storage Tiering
What you'll learn: How to optimize Pastebin storage costs by automatically moving older, rarely-accessed pastes to cheaper archival storage.
The Storage Cost Problem
In a system like Pastebin, not all pastes are accessed equally. New pastes often get heavy traffic in their first hours or days, but most eventually become dormant. Storing millions of cold pastes on expensive, high-performance storage (like SSD-backed databases or premium object storage) wastes money.
Hot vs Cold Storage
Think of hot storage as your kitchen counter—expensive space for items you use daily. Cold storage is your basement freezer—cheap, spacious, but slower to access.
Hot storage (S3 Standard, fast SSD):
- Fast retrieval (milliseconds)
- Higher cost per GB
- For recent or frequently accessed pastes
Cold storage (S3 Glacier, tape backup):
- Slower retrieval (minutes to hours)
- Much cheaper per GB (often 10× less)
- For old or rarely accessed pastes
Implementing Tiering
After a paste's retention period (say, 30 days of inactivity or 6 months since creation), automatically migrate it:
- Scheduled job scans the metadata database for old pastes
- Copy paste content from S3 Standard → S3 Glacier
- Update metadata to track new storage location and tier
- Delete from hot storage to free space
When a cold paste is requested:
- Initiate retrieval from Glacier (may take minutes)
- Optionally promote back to hot storage if accessed frequently
- Cache the result to avoid repeated slow fetches
Cost Savings
A paste system storing 10 TB might keep only 1 TB hot and 9 TB cold, reducing storage costs by 80%+ while maintaining excellent performance for active content.
Key Takeaway:
Hot vs cold storage tiering dramatically reduces costs by automatically moving inactive pastes to cheaper archival storage like S3 Glacier, accepting slower retrieval times for rarely accessed content.