Confidence-Based Routing
What You'll Learn
You'll learn how to use model confidence scores to intelligently decide when a cheaper model's answer is good enough and when to escalate to a more capable (and expensive) model.
The Core Idea
Confidence-based routing extends your dynamic router by examining how confident a model is in its response. Many models return probability scores or logit values alongside their outputs. When confidence is high, you accept the result. When it's low, you automatically re-route the query to a stronger model.
Think of it like a medical triage system: a nurse (small model) can handle routine cases confidently. But when uncertain about a diagnosis, they escalate to a specialist (large model). You only pay for the specialist when truly needed.
How It Works
After your initial model generates a response, inspect its confidence metrics:
- Token probability: Average or minimum probability across generated tokens
- API confidence scores: Some providers return explicit confidence values
- Semantic uncertainty: Variation in multiple sampled responses
Set a confidence threshold (e.g., 0.75). Responses above this threshold are returned directly. Below it triggers re-routing to your tier-2 model.
response = small_model.generate(query)
confidence = response.average_token_probability
if confidence < CONFIDENCE_THRESHOLD:
response = large_model.generate(query)
return response
This approach saves costs on queries the small model handles confidently while maintaining quality through automatic escalation.
Key Takeaway: Confidence-based routing uses model uncertainty signals to dynamically escalate only low-confidence queries to more capable models, optimizing both cost and quality.