Course contentsShow
← System Design
Lesson 1212 of 1,91930. Observability: MetricsFree lesson

Aggregation Before Collection

Pre-aggregating metrics at the source to reduce storage costs and improve query speed for common use cases.

Aggregation Before Collection

You'll learn how pre-aggregating metrics at the source dramatically reduces storage costs and speeds up queries.

The Core Idea

Instead of sending every individual metric data point to your centralized metrics system, you aggregate metrics locally at the source (application, service, or agent) before transmission. Rather than shipping thousands of individual request durations every second, you compute and send summaries like counts, sums, min/max, and percentiles right where the data originates.

Why This Matters

Imagine a high-traffic service recording 100,000 requests per second. Sending every latency measurement individually would generate 8.6 billion data points per day — overwhelming storage and making queries painfully slow.

With aggregation before collection, you instead send pre-computed statistics every 10 seconds: request count, average latency, 95th percentile, etc. Now you're transmitting 8,640 aggregated data points per day instead. That's a million-fold reduction.

Real-World Application

Your application maintains in-memory aggregators (histograms, counters) that accumulate metrics over a scrape interval. When your metrics system pulls or the push interval triggers, you send the aggregated summary and reset the local counters.

This is exactly what StatsD does — it receives raw measurements from applications, aggregates them over flush intervals (typically 10s), then forwards summarized metrics to Graphite or other backends. Similarly, Prometheus client libraries maintain local histograms that are scraped periodically.

The Trade-Off

You lose access to individual raw data points, but gain massive efficiency. For most monitoring use cases (tracking trends, alerting on thresholds), aggregated metrics are sufficient.

Key Takeaway: Aggregating metrics at the source before transmission drastically reduces storage requirements and query latency while preserving the statistical insights you need for monitoring.