Alert vs Notification
What you'll learn: You'll understand the critical difference between alerts that demand immediate action and notifications that simply inform, so you can reduce noise and respond effectively to real problems.
The Core Distinction
Not every message from your monitoring system deserves to wake someone up at 3 AM. Alerts are urgent, actionable signals that require immediate human intervention to prevent or mitigate user impact. Notifications are informational messages that provide awareness but don't require urgent response.
Think of it like this: An alert is a fire alarm demanding you evacuate now. A notification is a weather forecast telling you it might rain tomorrow—useful to know, but you'll check it when convenient.
When to Alert
Send an alert when:
- Your error budget is burning rapidly or nearly exhausted
- Users are actively experiencing degraded service
- A critical dependency has failed
- Immediate action can prevent or reduce customer impact
Alerts should always answer: "What do I need to do right now?"
When to Notify
Send a notification when:
- A batch job completed successfully
- Usage patterns show interesting trends
- Non-critical resources are approaching capacity (but not yet critical)
- Background maintenance finished
Notifications are for awareness, planning, and asynchronous review. They belong in Slack channels, email digests, or dashboards—not pagers.
The Alert Fatigue Problem
Treating everything as an alert trains your team to ignore notifications. If someone gets paged for non-urgent issues, they'll start missing the real emergencies buried in noise. Reserve alerts for genuine "someone must look at this immediately" situations.
Key Takeaway: Alerts require immediate action to prevent user impact; notifications provide useful information without urgency. Respecting this boundary prevents alert fatigue and ensures your team responds effectively when it truly matters.