Course contentsShow
← AI Engineering
Lesson 424 of 1,88611. RAG - Improving GenerationFree lesson

Confidence Scores and Thresholding

Use vector similarity scores to filter out low-confidence retrieval results before generation.

Confidence Scores and Thresholding

What you'll learn: How to use vector similarity scores to filter out low-quality retrieval results before they reach your LLM.

The Core Problem

When your retrieval system returns documents, not all matches are equally useful. Some chunks may be vaguely related but misleading; others might be completely off-topic. Sending poor matches to your LLM wastes context window space and can introduce irrelevant information that degrades answer quality.

What Are Confidence Scores?

Every vector similarity search returns a score for each retrieved document—typically a cosine similarity value between 0 and 1 (or -1 and 1, depending on the metric). Higher scores indicate stronger semantic matches between your query and the chunk.

Think of it like a search engine's relevance rating: a score of 0.9 might mean "highly relevant," while 0.3 suggests "weak connection at best."

Applying a Threshold

A threshold is a cutoff value you set. Any retrieved document scoring below this threshold gets discarded before generation.

Example: If you set a threshold of 0.7, only chunks with similarity ≥ 0.7 proceed to your prompt. This is like saying, "Only show me results I'm at least 70% confident about."

Why This Matters

  • Reduces noise: Prevents irrelevant context from confusing the LLM
  • Saves tokens: Fewer documents mean more efficient context usage
  • Improves accuracy: The LLM works with higher-quality information

Finding the Right Threshold

Start with a moderate value (e.g., 0.6–0.7) and evaluate output quality. Too high? You might filter out useful results. Too low? Junk gets through. Iteration and testing are key.

Key Takeaway: Confidence score thresholding acts as a quality gate, ensuring only sufficiently relevant chunks reach your LLM for generation.