Context Length and Compression Trade-offs
You'll learn why retrieved context frequently exceeds model token limits and when you need compression to make RAG systems work.
The Context Overflow Problem
When you retrieve documents in a RAG pipeline, you often face a frustrating dilemma: your reranked results might return 10-20 highly relevant chunks, but your LLM only has room for 3-5 of them. Every model has a context window — a hard limit on total input tokens (prompt + retrieved context + conversation history).
Think of it like packing a suitcase: you've gathered 20 essential items (retrieved chunks), but your bag (context window) only fits 8. You must either leave items behind or compress what you're bringing.
When Compression Becomes Necessary
You need compression when:
- Retrieved chunks exceed available tokens: After reserving space for your prompt template and generation, the context won't fit
- Cost considerations: Larger context windows cost more per API call
- Latency concerns: Processing 10,000 tokens takes longer than 2,000
- Quality over quantity: 15 mediocre chunks may perform worse than 3 compressed, highly-focused excerpts
The Trade-off Spectrum
You can handle overflow by:
- Truncating — drop lower-ranked chunks (loses information)
- Compressing — use an LLM to extract only relevant sentences (keeps signal, removes noise)
- Summarizing — condense each chunk before injecting (risks losing detail)
Each approach sacrifices something: completeness, computational cost, or precision.
Key Takeaway: Retrieved context often exceeds token budgets, forcing you to choose between truncation, compression, or summarization — each with distinct trade-offs in information retention, cost, and latency.