Context Window Constraints in RAG
What you'll learn: How the fixed token limits of LLMs determine how much retrieved information you can actually use in your RAG prompts.
The Token Budget Problem
Every LLM has a context window—a hard limit on how many tokens (roughly words or word pieces) it can process at once. For example, GPT-3.5-turbo has a 4,096 token limit, while GPT-4 offers up to 128,000 tokens depending on the version.
In RAG, your context window must accommodate:
- The system prompt (instructions for the model)
- The user's query
- All the retrieved documents you want to include
- Space for the model's response
This creates a crucial constraint: no matter how many documents your vector database retrieves, you can only pass a limited amount of content to the LLM.
The Practical Impact
Imagine your context window as a suitcase. You've packed:
- Instructions (10% of space)
- User question (5% of space)
- Response buffer (20% of space)
That leaves only ~65% for retrieved content. If each retrieved document is 500 tokens and your window is 4,000 tokens, you can realistically include only 4-5 documents—even if your retrieval step found 20 relevant ones.
Managing the Constraint
You must either:
- Retrieve fewer documents (top-3 instead of top-10)
- Truncate retrieved content (first 300 tokens per document)
- Choose models with larger windows (trade-off: cost and latency)
- Chunk more aggressively at index time (smaller, focused chunks)
Key Takeaway: Context window limits force you to be selective about which retrieved content makes it into your generation prompt—retrieval quality matters even more because you can't include everything.