Course contentsShow
← AI Engineering
Lesson 425 of 1,88611. RAG - Improving GenerationFree lesson

LLM-Based Relevance Filtering

Use a fast LLM to judge whether each retrieved chunk is relevant to the user's query.

LLM-Based Relevance Filtering

What you'll learn: How to use a lightweight LLM to automatically filter out irrelevant retrieved chunks before generation.

The Core Problem

You've retrieved documents using embeddings and maybe even reranked them, but some chunks still might not actually answer the user's question. Passing irrelevant context wastes tokens, confuses the generation model, and can lead to hallucinations or poor answers.

How LLM-Based Filtering Works

Instead of trusting retrieval scores alone, you send each retrieved chunk to a fast, inexpensive LLM (like GPT-3.5-turbo or a small local model) and ask: "Does this chunk contain information relevant to answering the user's question?"

The LLM returns a simple yes/no judgment or relevance score. You then filter out chunks marked as irrelevant before passing context to your final generation step.

Think of it like a quality control inspector: Before ingredients reach the chef (your generation model), an inspector (filtering LLM) checks each one and discards anything that doesn't belong in the recipe.

Why This Works

LLMs can understand semantic relevance in ways that vector similarity alone cannot. A chunk might have high embedding similarity but discuss a different aspect of the topic, use the term differently, or be outdated. The filtering LLM catches these mismatches by actually "reading" the content.

The Trade-off

This adds an extra LLM call per chunk (though you can batch them), but you save tokens in the generation phase and improve answer quality. Use a small, fast model for filtering to keep latency low.

Key Takeaway: LLM-based relevance filtering acts as a semantic quality gate, using a fast model to judge whether each retrieved chunk truly helps answer the query before passing it to generation.