Course contentsShow
← AI Engineering
Lesson 414 of 1,88611. RAG - Improving GenerationFree lesson

Context Window Management in RAG

Understand how to position retrieved documents within the prompt to maximize model attention and relevance.

Context Window Management in RAG

You'll learn how to strategically position retrieved documents within your prompt to maximize the language model's attention and response relevance.

The Core Challenge

Language models don't treat all parts of their input equally. Research shows they pay more attention to the beginning and end of prompts, while information in the middle can get "lost." This phenomenon—called the lost-in-the-middle problem—is critical when you're stuffing multiple retrieved documents into a RAG prompt.

Think of it like a sandwich: the model "tastes" the top slice (system instructions) and bottom slice (your question) most strongly, but the middle filling (your retrieved context) might blur together if not arranged carefully.

Strategic Positioning Patterns

Most Relevant First: Place your highest-ranked retrieved documents at the top of the context section, immediately after system instructions. The model sees these first and weights them heavily.

Question-Adjacent: Alternatively, position the most critical document right before the user's question at the bottom. This creates recency bias—the model "remembers" what it just read.

Sandwich Critical Content: For multiple documents, put highly relevant chunks at both the beginning and end of your context block, with less critical material in the middle.

Managing the Window

Your context window has finite space (measured in tokens). After retrieval and reranking, you might have 10 documents but room for only 5. Prioritize ruthlessly: only include what directly addresses the query. Use your reranking scores to cut low-confidence results before they even enter the prompt.

Key Takeaway: Position your most relevant retrieved documents at the start or end of your context section to combat the lost-in-the-middle problem and maximize model attention where it matters most.