Course contentsShow
← AI Engineering
Lesson 359 of 1,8869. Retrieval-Augmented Generation (RAG) - BasicsFree lesson

Context Compression On-the-Fly

Summarizing or extracting key sentences from retrieved documents before injection to save tokens.

Context Compression On-the-Fly

What you'll learn: How to shrink retrieved documents in real-time to fit more knowledge into your LLM's context window without losing critical information.

The Problem

You've retrieved 10 relevant documents from your vector database, but they contain 20,000 tokens combined—far too much for your context window or budget. Simply cutting them off means losing valuable information. Jamming them all in wastes tokens on repetitive or irrelevant details.

The Core Idea

Context compression on-the-fly means processing retrieved documents after retrieval but before prompt injection to extract only the most relevant parts. Think of it like highlighting a textbook: instead of photocopying entire chapters, you scan through and mark only the sentences that answer your specific question.

Two main approaches:

  1. Extractive summarization: Pull out key sentences or passages that directly relate to the user's query
  2. Abstractive summarization: Use a smaller LLM to generate concise summaries of each document

When to Use This

  • Your retrieved chunks are verbose or contain tangential content
  • You need to include more documents than your context window allows
  • Token costs are a concern and you're paying per input token
  • The query is specific enough that only portions of each document matter

The Trade-off

Compression adds latency (you're running an extra processing step) and may occasionally discard nuanced information. However, it can dramatically improve your effective retrieval count—fitting 10 compressed documents where only 3 full ones would fit.

Key Takeaway: Context compression on-the-fly lets you maximize your context window's value by keeping only query-relevant content from retrieved documents, trading a small processing overhead for better token efficiency and broader knowledge coverage.