Symptoms vs Causes
What you'll learn: How to alert on what users experience rather than why systems behave a certain way, keeping your monitoring focused on actual impact.
The Core Idea
When designing alerts, distinguish between symptoms (user-visible problems) and causes (underlying technical reasons). Alert on symptoms first because they tell you whether there's a problem users are experiencing. Causes explain why, but alerting on causes alone can flood you with noise about issues that haven't actually degraded user experience.
Symptoms: The "What Users Feel"
Symptoms are user-facing signals like:
- Slow page loads (high latency)
- Failed requests (elevated error rates)
- Service unavailable (complete outages)
These directly map to your SLIs—the metrics that represent your service quality promises.
Causes: The "Why It's Happening"
Causes are low-level technical indicators:
- CPU usage at 80%
- Memory consumption rising
- Database connection pool exhausted
- Disk I/O saturation
These metrics are important for investigation and diagnosis, but they don't always mean users are affected. A spike in CPU might be handled perfectly by autoscaling before anyone notices.
The Right Alert Strategy
Alert on symptoms to know when to respond. Use cause-based metrics in your dashboards and investigation tools to understand what to fix.
Think of it like a car: you want a dashboard light when the engine overheats (symptom affecting drivability), not just when coolant temperature reaches an arbitrary threshold that might be harmless.
Key Takeaway: Alert on user-facing symptoms like errors and latency that violate your SLOs, then investigate causes like CPU or memory during incident response—not the other way around.