Writing Custom PromQL Queries for Resource Analysis

In this lesson, you will learn how to write effective PromQL queries to transform raw metrics from cAdvisor and Node Exporter into actionable insights for troubleshooting Kubernetes performance.

The Power of Aggregation

Raw metrics are often too granular to be useful during an incident. When troubleshooting, you rarely want to look at a single container's CPU usage; you want to see the performance trends of an entire deployment. PromQL allows you to aggregate these metrics across dimensions using operators like sum, avg, and rate.

Analyzing CPU Throttling

A classic performance issue in Kubernetes is CPU throttling, which occurs when a container hits its limits.cpu. To detect this across a deployment, we combine the container_cpu_cfs_throttled_periods_total metric with aggregation:

sum by (pod) (
  rate(container_cpu_cfs_throttled_periods_total{namespace="production"}[5m])
) / 
sum by (pod) (
  rate(container_cpu_cfs_period_total{namespace="production"}[5m])
)

Breaking down the query:

1. rate(...[5m]): Calculates the per-second rate of throttled periods over the last five minutes. This transforms a counter (which only goes up) into a readable velocity.
2. sum by (pod): Groups the results so you can see which specific pods are experiencing the highest rate of throttling.
3. The Division: By dividing throttled periods by the total periods, we get a percentage, allowing you to easily identify pods that are consistently starved for CPU.

Best Practices for Troubleshooting

• Use rate() for Counters: Never run a raw counter in a graph; always use rate() or irate() to understand the current velocity of change.
• Leverage Labels: Use sum by (label_name) to pivot your data. If you suspect a bad deployment, sum by (pod_template_hash) can help you isolate the issue to a specific ReplicaSet.
• Time Windows: Use a window (e.g., [5m]) that is significantly larger than your scrape interval to ensure the query isn't overly sensitive to minor fluctuations in data collection.

By moving from raw data to calculated ratios, you can distinguish between a noisy application and a genuine resource bottleneck.

Key Takeaway: Use rate() to convert cumulative counters into velocity and aggregate via sum by to identify resource bottlenecks at the application or pod level.