Course contentsShow
← Cloud and Infrastructure
Lesson 1978 of 2,98942. Observability - MetricsFree lesson

High Cardinality and Label Design

Managing metric cardinality through proper label design and avoiding patterns that cause dimensional explosion.

High Cardinality and Label Design

What you'll learn: How to prevent Prometheus from drowning in time-series data by designing labels wisely and avoiding cardinality explosions.

Understanding High Cardinality

In Prometheus, every unique combination of label values creates a separate time series. High cardinality occurs when labels have many possible values, multiplying your series count exponentially. Think of it like a spreadsheet: adding a column with 1,000 unique values across 100 rows doesn't just add 1,000 cells—it can create 100,000 unique combinations when crossed with other dimensions.

This matters because Prometheus stores each time series in memory. Too many series = memory exhaustion, slow queries, and scrape failures.

Dangerous Label Patterns

Never use labels for:

  • User IDs or session IDs (millions of unique values)
  • Timestamps or full URLs (infinite possibilities)
  • Request IDs or transaction hashes (unbounded growth)
  • IP addresses (unless carefully controlled)

Good Label Design

Keep labels to bounded dimensions that help with aggregation:

  • Environment: {env="prod"} (3-5 values)
  • Service name: {service="api"} (dozens, not thousands)
  • HTTP status class: {status="5xx"} (not individual codes)
  • Geographic region: {region="us-east"} (limited set)

A common safe cardinality formula: if you have 100 services × 3 environments × 5 regions = 1,500 series. Manageable. But 100 services × 10,000 customer IDs? That's 1 million series—disaster.

The Fix

When you need granular data, use logs or traces instead. Metrics answer "how much/how many?", not "for whom specifically?". Aggregate first, drill down elsewhere.

Key Takeaway: Design labels with low, bounded cardinality by avoiding high-variance dimensions like IDs, keeping your metric count manageable and Prometheus performant.