Course contentsShow
← System Design
Lesson 1293 of 1,91932. Monitoring and AlertingFree lesson

Alert Context and Enrichment

Include relevant context (affected users, graphs, logs) in alerts to speed diagnosis without requiring dashboard lookups.

Alert Context and Enrichment

You'll learn how to embed diagnostic information directly into alerts so responders can act immediately without hunting through dashboards.

The Problem: Alerts That Leave You Guessing

When an alert fires at 3am, the worst scenario is receiving a bare message like "CPU high on service-x." The responder wakes up, opens their laptop, and now must:

  • Find the right dashboard
  • Identify which instances are affected
  • Check recent deploys
  • Look up current traffic patterns
  • Correlate with logs

This wastes precious minutes during an incident. Alert enrichment solves this by packaging critical context inside the alert itself.

What to Include in Alert Context

Think of an enriched alert like a complete patient chart handed to an emergency room doctor—not just "patient sick," but vitals, history, and current symptoms.

Include:

  • Affected scope: Which service instances, regions, or user segments
  • Current metrics: The actual value that triggered the alert (e.g., "CPU at 94%, threshold 85%")
  • Visual snapshots: Links to pre-filtered dashboard views or embedded graphs
  • Recent changes: Last deployment timestamp, config changes
  • Impact estimate: "Affecting 2,500 users in us-east-1"
  • Runbook link: Direct pointer to diagnostic steps
  • Related logs: Deep links to log queries for the time window and affected hosts

Implementation Pattern

# Example alert template with enrichment
alert: HighErrorRate
message: |
  Service: payment-api
  Error rate: 12.3% (threshold: 5%)
  Affected: us-west-2, 3 instances
  Impact: ~1,200 checkout attempts failing
  Graph: https://dash/errors?service=payment&last=30m
  Logs: https://logs/query?error=true&service=payment
  Runbook: https://wiki/runbook/payment-errors

Key Takeaway: Enriched alerts frontload all diagnostic context—metrics, graphs, logs, and runbook links—so responders can begin mitigation immediately without dashboard archaeology, dramatically reducing mean time to resolution (MTTR).