Course contentsShow
← System Design
Lesson 1217 of 1,91930. Observability: MetricsFree lesson

Sampling for Expensive Metrics

Using statistical sampling to collect high-cost metrics while maintaining acceptable accuracy for analysis.

Sampling for Expensive Metrics

What you'll learn: How to use statistical sampling to gather costly metrics without overwhelming your system or budget.

The Problem with Expensive Metrics

Some metrics are expensive to collect. Computing detailed percentiles across millions of requests, tracking complex business calculations per transaction, or capturing high-resolution histograms can strain CPU, memory, network bandwidth, and storage. If you naively collect every data point, you may spend more resources on observability than on serving users.

How Sampling Works

Sampling means you only record a fraction of events while still maintaining statistical validity. Instead of tracking every request's latency, you might sample 1 in 100 requests (1% sampling rate). With a large enough sample size, you can still accurately estimate percentiles, averages, and distributions for the entire population.

Analogy: Political polls don't survey every voter—they sample 1,000-2,000 people and extrapolate to millions. The math works when done correctly.

Implementation Approaches

  • Random sampling: Select events probabilistically (e.g., if random() < 0.01)
  • Reservoir sampling: Maintain a fixed-size sample from an unbounded stream
  • Adaptive sampling: Increase sample rate when detecting anomalies or errors (sample failures more aggressively than successes)

When to Sample

Sample expensive metrics like:

  • Detailed histogram buckets for every endpoint
  • Per-user or per-tenant fine-grained metrics
  • High-frequency metrics from hot paths
  • Metrics requiring complex computation (cryptographic hashes, external lookups)

Always collect critical counters and gauges at 100%—sampling works best for aggregatable, statistical metrics.

Key Takeaway: Sampling lets you balance observability costs with accuracy by collecting only a representative subset of expensive metric data, leveraging statistical properties to maintain reliable insights at a fraction of the resource cost.