In this lesson, you will learn how to optimize high-volume time-series data storage by grouping related readings into a single document using the Bucket Pattern.
The Problem: Index Bloat
Imagine you are building an IoT application where a temperature sensor reports data every second. If you store every single reading as an individual document in a collection, your indexes grow rapidly. Every new reading requires the database to update its index tree. Over millions of readings, the index size explodes, consuming vast amounts of RAM and slowing down performance significantly.
The Solution: The Bucket Pattern
Instead of one document per reading, we use the Bucket Pattern. You group a set of readings (e.g., one hour’s worth of data) into a single document. Each document acts as a "bucket" that holds an array of readings for that specific time range.
Consider this structure for a sensor:
{
"sensor_id": "temp-001",
"bucket_start": "2023-10-01T10:00:00Z",
"readings": [
{ "timestamp": "2023-10-01T10:00:01Z", "value": 22.5 },
{ "timestamp": "2023-10-01T10:00:02Z", "value": 22.6 }
],
"count": 2,
"sum": 45.1
}
Why it Works
By using this approach, your index only needs to track the sensor_id and the bucket_start field. Instead of 1,000 index entries for 1,000 readings, you only have one entry for the entire bucket.
When a new reading arrives, you use an upsert or an atomic update to push the new value into the readings array of the current bucket. This drastically reduces the write load on your index trees and keeps your "hot" working set small, allowing the database to perform significantly faster. Additionally, you can store pre-calculated values like sum or average inside the document, which saves compute time during read queries.
Key Takeaway: The Bucket Pattern reduces index size and write overhead by aggregating multiple related data points into a single document, effectively turning a "sea of documents" into a manageable set of data containers.