Alert Composition: AND, OR, and Absence Conditions
What you'll learn: How to combine multiple monitoring signals into smarter, more reliable alerts that cut through the noise.
The Noise Problem
Single-condition alerts often fire too aggressively. A CPU spike might trigger an alert, but was it actually a problem? Did users notice? Combining multiple signals creates alerts that reflect real issues, not just transient blips.
AND Conditions: Multiple Symptoms Together
Use AND to require multiple conditions before alerting. This catches scenarios where several things go wrong simultaneously—a strong signal that something truly matters.
Example: Alert when high error rate AND elevated latency AND low success rate all occur together. Any one alone might be noise; all three together indicates a real incident affecting users.
OR Conditions: Alternative Failure Modes
Use OR to catch different symptoms of the same underlying problem. Think of it as "any of these symptoms indicates trouble."
Example: Alert when database connection pool exhausted OR query timeout rate spikes OR connection errors increase. These are different manifestations of database distress.
Absence Conditions: The Silent Killer
Sometimes the absence of expected signals indicates failure. If a heartbeat stops or metrics disappear, that's often worse than bad metrics.
Example: Alert when no successful health checks received in 5 minutes. Silence can mean the service crashed completely—no metrics means no data, which is critically different from bad data.
Combining Them
Real-world alerts often mix these operators:
Alert when (high latency AND elevated errors) OR (no requests received in 10 minutes)
This catches both degraded performance and complete service death.
Key Takeaway: Compose alerts using AND (multiple symptoms), OR (alternative failure modes), and absence conditions (detecting silence) to create actionable alerts that fire only when real problems occur, dramatically reducing false positives and alert fatigue.