Course contentsShow
← AI Engineering
Lesson 354 of 1,8869. Retrieval-Augmented Generation (RAG) - BasicsFree lesson

Limiting Retrieved Context

Strategies for selecting top-k documents and truncating content to fit within token budgets.

Limiting Retrieved Context

What you'll learn: How to select and truncate retrieved documents so your RAG prompt fits within the LLM's token budget without losing critical information.

Why Limiting Matters

You've retrieved relevant documents from your vector database, but there's a problem: you can't send everything to the LLM. Context windows have hard limits (remember those constraints from lesson 332?), and even if they didn't, more tokens mean higher costs and slower responses.

Think of it like packing a suitcase for a flight—you want to bring everything useful, but you're constrained by weight limits. You must choose wisely.

Top-K Document Selection

The simplest strategy is top-k retrieval: take only the k most relevant documents based on similarity scores. If your vector search returns 50 documents but your budget allows only 5, you keep the top 5 and discard the rest.

# You already know how to get scored results
# Now limit to top-k
k = 5
top_docs = retrieved_results[:k]  # Take only first k results

Content Truncation Strategies

Even after selecting top-k documents, individual chunks might be too long. You have options:

  • Hard truncation: Cut each document at a fixed token count
  • Proportional allocation: Distribute tokens across documents (e.g., 200 tokens each from 5 docs = 1000 tokens total)
  • Priority-based: Give more tokens to higher-ranked documents
  • Sentence-boundary truncation: Cut at complete sentences to maintain readability

Practical Considerations

Always leave headroom for your system prompt, user query, and the LLM's response. If your context window is 4096 tokens, budget perhaps 2500 for retrieved context, 500 for prompts, and 1096 for generation.

Key Takeaway: Limiting retrieved context through top-k selection and smart truncation ensures your RAG system stays within token budgets while maximizing the quality of information passed to the LLM.