Alert Fundamentals: Symptoms vs. Causes
You'll learn why effective alerting focuses on what users experience, not what breaks internally.
The Core Principle
When your system has problems, there are two types of signals you could alert on:
- Symptoms: What users actually experience (slow page loads, errors, failed transactions)
- Causes: What broke internally (high CPU, disk full, service restart)
The golden rule: alert on symptoms, debug with causes.
Why Alert on Symptoms?
Imagine your car dashboard. It warns you when the engine overheats (symptom affecting your ability to drive), not when the fan belt is slightly loose (internal cause that might matter). You care about "can I drive safely?" not "is every component perfect?"
Similarly, alert when:
- Error rates spike (users see failures)
- Response times exceed SLOs (users experience slowness)
- Availability drops (users can't access the service)
Don't alert when:
- CPU hits 80% (might be fine if response times are normal)
- One replica crashes (if others handle the load)
- Cache miss rate increases (unless it impacts user experience)
Reducing Alert Noise
Cause-based alerts create noise because not every internal issue affects users. Maybe your database connection pool is maxed out, but requests are still fast. Alerting on CPU across all servers means waking someone up when it doesn't matter.
Symptom-based alerts are actionable: if users are impacted, you investigate. Then you look at internal causes (metrics, logs, traces) to understand why.
The Connection to Golden Signals
This philosophy aligns with the Four Golden Signals: latency, traffic, errors, and saturation describe user-facing symptoms, guiding you to problems that actually matter.
Key Takeaway: Alert on user-visible symptoms to reduce noise and focus investigations on real impact; use internal metrics as diagnostic tools, not primary alerts.