Automating Alerts for Error Budget Exhaustion

In this lesson, you will learn how to configure intelligent alerting thresholds that notify your engineering team only when your error budget is at significant risk of exhaustion, preventing "alert fatigue."

The Problem with Naive Alerting

Standard alerts often trigger on instantaneous spikes, which can be noisy and misleading. An error budget—the amount of unreliability your service is allowed over a period (e.g., 30 days)—is meant to be consumed at a steady rate. If you consume your entire budget in five minutes, you have a critical incident; if you consume it at a slow, constant trickle, you have a long-term reliability trend to address. We need to alert based on the burn rate of the budget.

Configuring Burn Rate Alerts

Instead of alerting on a static error count, we alert on how quickly the budget is being consumed. A common practice is to use Prometheus recording rules to calculate the burn rate over specific windows (e.g., 1 hour vs. 6 hours).

To notify the team only when the risk is high, we set thresholds based on the SLO burn rate:

1. Critical Alert (Fast Burn): Trigger if the budget is consumed at 14.4x the allowed rate over the last 1 hour. This indicates a major incident that will exhaust the monthly budget in 2 days.
2. Warning Alert (Slow Burn): Trigger if the budget is consumed at 6x the allowed rate over the last 6 hours. This warns the team that, if the current trend continues, the entire monthly budget will be gone within a week.

groups:
- name: error-budget-alerts
  rules:
  - alert: HighErrorBudgetBurn
    expr: |
      (job:slo_errors_per_window:rate1h > 14.4)
      OR
      (job:slo_errors_per_window:rate6h > 6)
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Error budget burning too fast"
      description: "Service is burning its error budget at a rate that will exhaust it prematurely."

By focusing on these burn rates, you ensure that the on-call engineer is only interrupted for meaningful events that actually threaten the reliability goals defined by your SLOs, rather than minor transient fluctuations.

Key Takeaway: Configure alerts based on error budget burn rates rather than raw error counts to ensure you only notify the team when the long-term reliability objective is genuinely at risk.