In this lesson, you will learn how to prevent severe performance degradation in time-series systems caused by high-cardinality tag dimensions.
The Cardinality Trap
In metric-based systems, data is often stored using tags (metadata) to categorize values, such as server_id, user_id, or request_id. Cardinality refers to the total number of unique combinations of these tag values.
A "high-cardinality" problem occurs when you use dimensions with an unbounded range of values—like an ip_address or a transaction_uuid—as a tag. Because storage engines (like Prometheus or InfluxDB) create an index entry for every unique series (the combination of measurement name and tag set), high cardinality forces the engine to manage millions of distinct index keys. This consumes massive amounts of RAM and causes the "churn" that leads to query timeouts and storage bloat.
Strategies for Control
To prevent this, you must treat your tag schema as a strictly managed interface.
1. The Dimensionality Limit
Never use unique identifiers (IDs) as tags. Instead of:
cpu_usage{instance_id="user_unique_uuid_123"}
Map these into a higher-level attribute or move them to your application logs where they belong. Only use tags for dimensions that have a finite, relatively small number of values (e.g., region, environment, status_code).
2. Dropping High-Cardinality Tags at Ingestion
If a rogue process starts sending high-cardinality metrics, your ingestion pipeline should drop them before they hit the index. In a SQL-based time-series setup, you can enforce this with a validation layer:
-- Conceptual validation: Avoid indexing high-cardinality columns
-- Instead of indexing every unique user, aggregate by group.
SELECT region, AVG(cpu_load)
FROM metrics
WHERE timestamp > NOW() - INTERVAL '1 hour'
GROUP BY region;
If you find your index size growing linearly with the number of incoming events, you have a cardinality leak. Always audit your schema to ensure that your tag combinations are predictable and bounded. If a query requires searching by a high-cardinality field, perform that search against a relational store or log index, and keep your metric time-series data focused on aggregatable dimensions.
Key Takeaway: High cardinality is the primary cause of time-series database collapse; never use unique IDs as tags, and ensure your tag-based dimensions represent finite categories, not specific instances.