Threshold Selection and Tuning
What You'll Learn
You'll learn how to set alert thresholds that reflect real business impact rather than arbitrary technical numbers.
The Problem with Arbitrary Thresholds
Many teams set alert thresholds based on gut feeling or round numbers: "Let's alert when CPU hits 80%" or "Page if error rate exceeds 5%." But why 80%? Why 5%? Without business context, these numbers create either too many false alarms or miss real problems.
Ground Thresholds in SLOs and Data
Use your SLO budget as the foundation. If your error budget allows 0.1% errors over a month, your alert threshold should fire before you exhaust that budget—not at some arbitrary percentage that sounds scary.
Example approach:
- Your SLO: 99.9% availability (0.1% error budget monthly)
- Historical data shows normal error rate: 0.01%–0.03%
- Threshold: Alert when errors exceed 0.05% for 10 minutes (well above normal, well below budget exhaustion)
Historical Data Tells the Truth
Analyze your metrics over weeks or months. What's the normal range? What preceded past incidents? Set thresholds that:
- Sit comfortably above normal noise
- Trigger early enough to prevent SLO violations
- Align with burn rate (how fast you're consuming error budget)
Think of it like a fuel gauge: you don't wait until empty to refuel, but you also don't panic at three-quarters full. Your threshold is the "time to find a gas station" point based on how fast you're burning fuel.
Iterate and Refine
No threshold is perfect on day one. Review alerts monthly: too many false positives? Tighten. Missed an outage? Loosen or add complementary signals.
Key Takeaway: Set alert thresholds based on SLO budgets and historical patterns, not arbitrary numbers—thresholds should reflect real business impact and give you time to act before customers notice.