Aggregation and Roll-Ups
What you'll learn: How metrics are aggregated over time windows and downsampled for long-term storage efficiency.
The Storage Problem
Raw metrics arrive continuously—imagine thousands of servers reporting CPU usage every second. If you store every data point indefinitely, you'll drown in data. Within months, you'd need petabytes just to remember what happened last year. This is where aggregation and roll-ups become essential.
What is Aggregation?
Aggregation means combining multiple raw metric values into a single summary value over a time window. Instead of storing 60 individual counter increments per minute, you store one aggregated sum per minute.
For example:
- Counter: Sum all increments in a 5-minute window
- Gauge: Take the average, min, or max value over 1 hour
- Histogram: Combine buckets from multiple measurements
What are Roll-Ups?
Roll-ups (or downsampling) mean progressively aggregating older data into coarser time windows. It's like zooming out on a map—you lose detail but gain perspective.
Typical strategy:
- Raw data: Keep 1-second resolution for 1 day
- 5-minute roll-ups: Keep for 30 days
- 1-hour roll-ups: Keep for 1 year
- 1-day roll-ups: Keep indefinitely
You sacrifice granularity for storage efficiency. You can't analyze a 3-second spike from six months ago, but you can still see daily trends from three years back.
The Trade-Off
Aggregation is lossy. Once you've averaged gauge values over an hour, you can't recover the individual spikes. Choose your aggregation functions wisely based on what questions you'll need to answer later.
Key Takeaway: Aggregation and roll-ups let you retain long-term metric history by progressively reducing resolution over time, trading granular detail for manageable storage costs.