Setting Alerting Rules for Cluster Health

In this lesson, you will learn how to transition from simply visualizing telemetry data to proactively detecting cluster issues by defining Prometheus alerting rules.

From Monitoring to Alerting

So far, we have focused on scraping metrics and visualizing them in Grafana. However, staring at dashboards 24/7 is not scalable. Prometheus alerting rules bridge the gap between "observing" and "acting" by evaluating PromQL expressions at set intervals. When a expression evaluates to true for a specified duration, Prometheus fires an alert, which can then be routed to tools like Slack, PagerDuty, or email via the Alertmanager.

Defining an Alerting Rule

Alerting rules are defined in YAML files, typically stored in a PrometheusRule custom resource if you are using the Prometheus Operator. A rule consists of an alert name, a expr (the PromQL query), and a for duration (to avoid flapping due to momentary spikes).

Consider this example, which alerts you if a node has been experiencing high CPU utilization for more than five minutes:

groups:
  - name: cluster-health.rules
    rules:
      - alert: HighNodeCPU
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 90% for more than 5 minutes."

Best Practices for Healthy Alerts

1. Use the for field: Always include a duration (e.g., 5m). This prevents "flapping" alerts, which occur when a metric hovers right at the threshold and triggers a barrage of notifications.
2. Prioritize Severity: Use labels like severity: critical or severity: warning to help your team prioritize responses.
3. Be Specific with Annotations: Include context in the description field. Using template variables like {{ $labels.instance }} ensures that the person receiving the alert knows exactly which component in your cluster is failing without needing to dig into Prometheus immediately.

By configuring these rules, you shift your cluster management from reactive firefighting to a managed, automated observability workflow.

Key Takeaway: Alerting rules transform passive metric collection into active cluster oversight by automating the detection of performance thresholds and providing actionable context for incident response.