Recording Rules for Performance
What you'll learn: How to use recording rules to speed up expensive queries by pre-computing their results at scrape time.
The Performance Problem
When you query Prometheus with complex PromQL expressions—especially ones involving heavy aggregations, multiple labels, or long time ranges—the query can take seconds or even timeout. Imagine calculating the average CPU usage across 1,000 pods, grouped by deployment, every time someone loads a dashboard. That's expensive!
What Are Recording Rules?
Recording rules are pre-computed queries that run at regular intervals (typically at scrape time) and store their results as new time-series metrics. Instead of computing sum(rate(http_requests_total[5m])) by (job) on-demand every time, you run it once per evaluation cycle and save the result as a simpler metric like job:http_requests:rate5m.
Think of it like batch-cooking meals on Sunday: you do the expensive prep work once, then just reheat throughout the week.
How They Work
Recording rules live in your Prometheus configuration and evaluate automatically. They transform complex expressions into efficient, pre-aggregated metrics that dashboards and alerts can query instantly. You define them in rule files:
groups:
- name: performance_rules
interval: 30s
rules:
- record: job:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (job)
Now queries can use job:http_requests:rate5m directly—no expensive rate calculations at query time.
When to Use Recording Rules
Use them for:
- Frequently accessed dashboard queries
- Complex aggregations across high-cardinality metrics
- Queries used in multiple alerts
- Reducing load on the Prometheus server during peak usage
Key Takeaway: Recording rules pre-compute expensive queries and store results as simple metrics, dramatically reducing query-time overhead and improving dashboard responsiveness at scale.