Course contentsShow
← AI Engineering
Lesson 1202 of 1,88629. Cost OptimizationFree lesson

Confidence-Based Routing

Use model confidence scores to decide whether to use results or route to a more capable model.

Confidence-Based Routing

What You'll Learn

You'll learn how to use model confidence scores to intelligently decide when a cheaper model's answer is good enough and when to escalate to a more capable (and expensive) model.

The Core Idea

Confidence-based routing extends your dynamic router by examining how confident a model is in its response. Many models return probability scores or logit values alongside their outputs. When confidence is high, you accept the result. When it's low, you automatically re-route the query to a stronger model.

Think of it like a medical triage system: a nurse (small model) can handle routine cases confidently. But when uncertain about a diagnosis, they escalate to a specialist (large model). You only pay for the specialist when truly needed.

How It Works

After your initial model generates a response, inspect its confidence metrics:

  • Token probability: Average or minimum probability across generated tokens
  • API confidence scores: Some providers return explicit confidence values
  • Semantic uncertainty: Variation in multiple sampled responses

Set a confidence threshold (e.g., 0.75). Responses above this threshold are returned directly. Below it triggers re-routing to your tier-2 model.

response = small_model.generate(query)
confidence = response.average_token_probability

if confidence < CONFIDENCE_THRESHOLD:
    response = large_model.generate(query)

return response

This approach saves costs on queries the small model handles confidently while maintaining quality through automatic escalation.

Key Takeaway: Confidence-based routing uses model uncertainty signals to dynamically escalate only low-confidence queries to more capable models, optimizing both cost and quality.