Alert Context and Enrichment
You'll learn how to embed diagnostic information directly into alerts so responders can act immediately without hunting through dashboards.
The Problem: Alerts That Leave You Guessing
When an alert fires at 3am, the worst scenario is receiving a bare message like "CPU high on service-x." The responder wakes up, opens their laptop, and now must:
- Find the right dashboard
- Identify which instances are affected
- Check recent deploys
- Look up current traffic patterns
- Correlate with logs
This wastes precious minutes during an incident. Alert enrichment solves this by packaging critical context inside the alert itself.
What to Include in Alert Context
Think of an enriched alert like a complete patient chart handed to an emergency room doctor—not just "patient sick," but vitals, history, and current symptoms.
Include:
- Affected scope: Which service instances, regions, or user segments
- Current metrics: The actual value that triggered the alert (e.g., "CPU at 94%, threshold 85%")
- Visual snapshots: Links to pre-filtered dashboard views or embedded graphs
- Recent changes: Last deployment timestamp, config changes
- Impact estimate: "Affecting 2,500 users in us-east-1"
- Runbook link: Direct pointer to diagnostic steps
- Related logs: Deep links to log queries for the time window and affected hosts
Implementation Pattern
# Example alert template with enrichment
alert: HighErrorRate
message: |
Service: payment-api
Error rate: 12.3% (threshold: 5%)
Affected: us-west-2, 3 instances
Impact: ~1,200 checkout attempts failing
Graph: https://dash/errors?service=payment&last=30m
Logs: https://logs/query?error=true&service=payment
Runbook: https://wiki/runbook/payment-errors
Key Takeaway: Enriched alerts frontload all diagnostic context—metrics, graphs, logs, and runbook links—so responders can begin mitigation immediately without dashboard archaeology, dramatically reducing mean time to resolution (MTTR).