Threshold-Based Alerting
What you'll learn: How to configure alerts that fire when specific metrics cross predefined static or dynamic boundaries in your LLM system.
The Core Idea
Threshold-based alerting triggers notifications when monitored metrics exceed acceptable limits. You define boundaries for critical metrics—like latency, error rates, token usage, or cost—and your monitoring system alerts you when they're breached.
Think of it like a smoke detector: you set a threshold (smoke concentration), and when that level is crossed, the alarm sounds. You're not constantly watching; the system watches for you.
Static vs. Dynamic Thresholds
Static thresholds are fixed values you set based on requirements or experience:
- Latency > 3 seconds
- Error rate > 2%
- Hourly token usage > 1 million
- Daily cost > $500
Dynamic thresholds adapt based on historical patterns and context:
- Latency > 2× the 7-day average
- Error rate > mean + 3 standard deviations
- Token usage > 150% of yesterday's peak hour
- Cost growth > 20% week-over-week
Static thresholds work well for hard business requirements. Dynamic thresholds catch anomalies in systems with natural variation (weekday vs. weekend traffic, seasonal patterns).
Common Metrics to Alert On
Based on your instrumentation (from lessons 1232-1241), you should alert on:
- Latency: P95/P99 response times crossing SLA bounds
- Error rates: Failed requests, timeouts, rate limit hits
- Token usage: Sudden spikes indicating inefficient prompts or abuse
- Cost metrics: Budget overruns or unexpected expense acceleration
- Quality degradation: Falling confidence scores or increased retries
Implementation Approach
Combine your metric collection with alert policies. Set multiple severity levels (warning vs. critical) and appropriate notification channels (email, Slack, PagerDuty) based on impact.
Key Takeaway: Threshold-based alerting transforms passive monitoring into active protection by automatically notifying you when metrics indicate problems, using either fixed limits for known requirements or adaptive boundaries for detecting unusual patterns.