Contextual Compression with LLMs
What you'll learn: How to use a language model to intelligently filter retrieved documents, keeping only the parts that actually answer your query.
The Problem: Too Much Irrelevant Context
When you retrieve documents in RAG, you often get back entire chunks that contain relevant information but also include a lot of noise. Imagine searching for "What is the capital of France?" and getting back a 500-word paragraph about French geography, history, and culture—the answer "Paris" is buried somewhere inside.
This wastes precious context window space and can distract the generation LLM with irrelevant details.
What Is Contextual Compression?
Contextual compression means sending each retrieved document through an LLM with a simple instruction: "Extract only the portions of this text that are relevant to answering the user's question."
Think of it like hiring an editor who reads through a stack of articles and highlights only the sentences that matter for your specific question, discarding everything else.
How It Works
- Retrieve documents normally using vector search or hybrid search
- Compress each document by prompting an LLM: "Given the query '{query}', extract only relevant excerpts from: {document}"
- Pass only the compressed results to your final generation step
The compression LLM acts as a relevance filter, dramatically reducing token count while improving signal-to-noise ratio.
Why It Helps
- Reduces context length so you can fit more truly relevant information
- Improves answer quality by removing distractions
- Preserves exact wording from source documents (unlike full summarization)
Key Takeaway: Contextual compression uses an LLM as a smart filter to extract only query-relevant portions from retrieved documents, improving both efficiency and answer quality in RAG systems.