LLM-Based Context Compression
What you'll learn: How to use smaller language models to intelligently compress retrieved passages, keeping what matters and discarding the noise.
The Core Idea
You've retrieved several relevant documents from your vector database, but together they're too long to fit comfortably in your context window—or they contain a lot of fluff that dilutes the important information. LLM-based context compression uses a small, fast language model to read through these passages and extract only the sentences or phrases that directly answer your user's question.
Think of it like hiring an assistant to read a stack of reports and highlight only the sentences relevant to your specific question, rather than forcing you to read everything.
How It Works
Instead of passing all retrieved chunks directly to your main LLM, you first route them through a compression step:
- Input: Your user query + all retrieved passages
- Compression LLM: A small model (like GPT-3.5-Turbo or a fine-tuned T5) reads each passage
- Instruction: "Extract only the sentences relevant to answering: [user query]"
- Output: Compressed context containing only essential information
- Final generation: Pass compressed context to your main LLM
Benefits Over Extractive Summarization
Unlike simple extractive methods (lesson 399) that use keywords or sentence scoring, LLM-based compression understands semantic relevance. It can identify that "The company's revenue grew 40% annually" is relevant to "How fast did the business expand?" even without keyword overlap.
Key Takeaway: LLM-based context compression uses smaller language models as intelligent filters, reducing token usage and improving response quality by keeping only query-relevant information from retrieved passages.