Setting Thresholds and Alert Policies
What You'll Learn
You'll learn how to design alerting rules for drift and anomaly detection that catch real issues without flooding your team with false alarms.
The Alert Balance Challenge
After detecting drift in your LLM system (via statistical methods, embeddings, or performance metrics from previous lessons), you need to decide: when should this trigger an alert? Set thresholds too sensitive, and you'll get alert fatigue from false positives. Set them too loose, and you'll miss critical degradation.
Designing Effective Alert Policies
Think of alerting like a smoke detector. You want it sensitive enough to catch real fires early, but not so sensitive it screams every time you toast bread.
Key Principles
1. Use Multiple Thresholds
- Warning threshold: Early signal that something might be wrong (e.g., 15% drift increase)
- Critical threshold: Definite problem requiring immediate action (e.g., 40% drift)
2. Time Windows Matter Don't alert on a single anomalous request. Use sustained periods:
- "Alert if embedding drift > 0.3 for 10+ consecutive minutes"
- "Critical if latency P99 > 5s for 3 out of 5 evaluation windows"
3. Context-Aware Thresholds Different scenarios need different sensitivity:
- Production traffic: stricter thresholds
- Canary deployments: looser, learning mode
- Low-traffic periods: higher tolerance for variance
4. Rate of Change Sometimes how fast metrics change matters more than absolute values. A sudden 20% spike in token usage is more concerning than a gradual monthly increase of 30%.
Practical Guideline
Start conservative (fewer alerts), monitor for 2-4 weeks, then tighten based on observed false negative patterns. Always tie alerts to actionable responses.
Key Takeaway: Effective alerting balances sensitivity with specificity using multi-level thresholds, time windows, and context-awareness—aim to catch meaningful degradation while preserving your team's attention for real issues.