In this lesson, you will learn how to mathematically forecast your disk requirements by balancing ingestion frequency, data cardinality, and retention windows.
The Storage Equation
To calculate how much storage a metric system requires, you must look beyond raw ingestion rates and account for the overhead of your chosen storage engine. The basic capacity model follows this formula:
Storage = (Metrics_per_Second * Bytes_per_Data_Point * Retention_Seconds) * (1 + Compression_Overhead)
- Metrics per Second: The total throughput of all your incoming data streams.
- Bytes per Data Point: This includes the timestamp (usually 8 bytes) and the value (typically 4–8 bytes), plus metadata overhead.
- Retention Seconds: How long the data persists before it is pruned or downsampled.
- Compression Overhead: If you are using delta-of-delta encoding or similar schemes, this acts as a multiplier (or rather, a divisor) for your raw size.
Factoring in Cardinality
A common mistake in capacity planning is ignoring cardinality. If you have one CPU metric but your system tags it by host_id, container_id, and region, your "one metric" is actually thousands of distinct time-series.
If your ingestion rate is 10,000 data points per second, and each data point occupies roughly 20 bytes (after metadata overhead), your growth rate is:
10,000 * 20 = 200,000 bytes/sec
200,000 * 86,400 seconds/day ≈ 17.28 GB per day.
Strategic Forecasting
When planning for a one-year horizon, do not simply multiply the daily rate by 365. You must integrate your downsampling strategies. If you keep high-resolution data for 7 days and downsampled (e.g., 1-minute averages) data for the remaining 358 days, your capacity forecast becomes:
-- Conceptual storage calculation
SELECT
(7 * (raw_ingress_per_day)) +
(358 * (downsampled_ingress_per_day)) AS total_annual_storage_bytes
FROM capacity_baseline;
As your cardinality grows—due to horizontal scaling of your application infrastructure—you must apply a "growth buffer" (typically 20–30%) to your forecast to avoid unexpected disk exhaustion during peak traffic events.
Key Takeaway: Precise capacity planning requires mapping your unique cardinality and downsampling policies against your raw ingestion rate to create a multi-tiered storage forecast.