Designing Observability Systems for Proactive Alerting

In this lesson, you will learn how to design observability systems that alert on user-facing symptoms of failure, shifting your focus away from noisy infrastructure component metrics.

Symptom-Based vs. Infrastructure-Based Alerting

In complex Kubernetes environments, it is tempting to alert on every high CPU usage or low memory metric. However, these are often "causes" rather than "symptoms." If a pod has high CPU, but your users are experiencing zero impact, an alert only generates toil and alert fatigue.

True observability for reliability focuses on symptoms: the user’s experience. If a user cannot checkout, cannot log in, or receives 500 errors, that is a symptom of failure. By alerting on symptoms, we ensure our team only responds to issues that actively burn through the Error Budgets we established in previous lessons.

Designing the Proactive Alert

Instead of monitoring node_memory_usage, we monitor SLIs (Service Level Indicators) like the success rate or latency of our primary service endpoints. We use Prometheus recording rules to aggregate these into burn rates, alerting us only when the rate of error consumption threatens our SLO.

Consider this PrometheusRule designed to alert on latency symptoms:

groups:
- name: latency-alerts
  rules:
  - alert: HighLatencySymptom
    expr: |
      (
        sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m])) 
        / 
        sum(rate(http_request_duration_seconds_count[5m]))
      ) < 0.95
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High latency detected: 95% of requests are slower than 500ms"

In this example, we aren't looking at why the pod is slow (infrastructure); we are alerting because the system is failing its promise to the user (symptom). If the application slows down but stays within our SLO threshold, the team is not interrupted. This keeps our focus on reliability engineering rather than fire-fighting infrastructure "noise."

By decoupling the alert from the infrastructure, you create a system that scales with your architecture and preserves the sanity of your on-call engineers.

Key Takeaway: Proactive alerting should focus on user-facing symptoms defined by your SLIs, ensuring you only wake up for incidents that impact your Service Level Objectives.