In this lesson, you will learn how to identify and prune high-cardinality label sets to prevent Prometheus performance degradation and storage bloat.
Understanding Cardinality
In Prometheus, cardinality refers to the number of unique combinations of label key-value pairs for a given metric name. Each unique combination creates a new time series in your database.
If you have a metric like http_requests_total and you attach a user_id or request_id as a label, the number of series explodes. For example, if you have 10 instances of an application and each handles 1,000 unique users per minute, you are suddenly generating 10,000 unique time series for just one metric. This "high cardinality" forces Prometheus to use massive amounts of RAM for indexing and significantly slows down query performance.
Identifying and Mitigating Bloat
To keep your telemetry pipeline healthy, you must avoid unbounded labels. A common mistake is using labels for data that is not intended for aggregation, such as request IDs, email addresses, or timestamps.
1. Identify High-Cardinality Metrics
You can use the built-in Prometheus status page at /-/status/tsdb to find which metrics have the highest number of active series. Look for metrics that grow linearly over time; these are often the primary offenders.
2. Relabeling to the Rescue
You can use Prometheus relabel_configs to drop or modify labels before they are ingested. If you find a label causing issues, you can remove it using the labeldrop action:
# In your Prometheus scrape_config
scrape_configs:
- job_name: 'my-app'
metric_relabel_configs:
- action: labeldrop
regex: 'request_id|user_email'
By dropping these high-cardinality labels at the source, you ensure that Prometheus only stores the aggregated metrics that are actually necessary for your alerts and dashboards. Always prioritize labels that represent fixed categories (like environment, cluster, or service_name) rather than unique identifiers.
Key Takeaway: Cardinality is the product of label value combinations; keep your metrics lean by dropping high-variance labels at the ingestion stage to protect your Prometheus memory and disk performance.