Alert Severity Levels and Response Expectations
You'll learn: How to classify alerts into severity tiers that dictate clear, realistic response expectations for your on-call teams.
The Core Idea
Not all alerts demand the same urgency. A database outage affecting every customer requires immediate action, while a single non-critical service experiencing elevated latency might wait until business hours. By defining severity levels—typically critical, warning, and info—you create a shared vocabulary that tells responders exactly how fast and how intensely they must act.
Think of it like emergency triage in a hospital: the patient with a heart attack (critical) gets immediate attention, someone with a sprained ankle (warning) waits for the next available doctor, and a routine check-up (info) is scheduled for later.
The Three Standard Tiers
Critical: System-wide impact or user-facing service down. Response required immediately (within minutes), often waking someone up. Example: payment processing completely broken.
Warning: Degraded performance or potential escalation. Response expected during business hours or within a few hours. Example: cache hit rate dropped 20%, API latency at 95th percentile rising.
Info: Context-enriching notifications with no immediate action required. Often used for audit trails or proactive awareness. Example: deployment completed successfully, autoscaling triggered.
Why This Matters
Without clear severity definitions, teams suffer from "alert fatigue"—everything feels urgent, so nothing is. Worse, responders waste time deciding how urgent something is instead of fixing it. Explicit response-time expectations (SLOs for alerting itself) prevent both under-reaction to real incidents and burnout from false urgency.
Key Takeaway: Define severity tiers with explicit response-time expectations so your team knows instantly whether to jump out of bed or address it Monday morning—reducing confusion, improving incident response, and preserving team wellbeing.