Alert Context: Including Diagnostic Information
What You'll Learn
You'll learn how to design alerts that include essential diagnostic information, enabling faster incident response and resolution.
The Problem with Context-Free Alerts
Imagine receiving a fire alarm that simply says "Fire detected" with no information about where the fire is, how big it is, or what floor you're on. That's what many alerts feel like to on-call engineers: they signal a problem but leave responders scrambling to gather basic context before they can even begin diagnosis.
Building Context-Rich Alerts
An actionable alert should answer three questions immediately:
- What's broken? (You already know this from earlier lessons)
- Where do I look? (Links to relevant dashboards, logs, traces)
- What do I do? (Links to runbooks or recent changes)
Essential Context Elements
Dashboard Links: Include direct links to pre-filtered dashboards showing the relevant metrics. If CPU is spiking, link to the CPU dashboard for that specific service and time window.
Runbook References: Link to or embed the URL of the runbook that addresses this specific alert. Your 2 AM self will thank you.
Recent Changes: Include references to recent deployments, configuration changes, or infrastructure modifications. Many incidents correlate with recent changes.
Query Templates: Provide ready-to-run queries for logs or traces that filter to the affected service and timeframe.
Affected Scope: Include labels or tags showing which instances, regions, or customers are impacted.
Example Context Block
Alert: API latency p99 > 500ms
Dashboard: https://dash.example.com/api-latency?service=checkout&last=30m
Runbook: https://wiki.example.com/runbooks/high-latency
Recent Deploy: v2.3.1 deployed 15 minutes ago by @alice
Query: service:checkout AND status:error AND timestamp:[now-30m TO now]
Key Takeaway: Context-rich alerts transform a wake-up call into a head start—include direct links to dashboards, runbooks, and recent changes so responders can diagnose and act immediately rather than hunting for information.