Course contentsShow
← AI Engineering
Lesson 332 of 1,8869. Retrieval-Augmented Generation (RAG) - BasicsFree lesson

Context Window Constraints in RAG

Understanding how LLM token limits affect how much retrieved content can be included in generation prompts.

Context Window Constraints in RAG

What you'll learn: How the fixed token limits of LLMs determine how much retrieved information you can actually use in your RAG prompts.

The Token Budget Problem

Every LLM has a context window—a hard limit on how many tokens (roughly words or word pieces) it can process at once. For example, GPT-3.5-turbo has a 4,096 token limit, while GPT-4 offers up to 128,000 tokens depending on the version.

In RAG, your context window must accommodate:

  • The system prompt (instructions for the model)
  • The user's query
  • All the retrieved documents you want to include
  • Space for the model's response

This creates a crucial constraint: no matter how many documents your vector database retrieves, you can only pass a limited amount of content to the LLM.

The Practical Impact

Imagine your context window as a suitcase. You've packed:

  • Instructions (10% of space)
  • User question (5% of space)
  • Response buffer (20% of space)

That leaves only ~65% for retrieved content. If each retrieved document is 500 tokens and your window is 4,000 tokens, you can realistically include only 4-5 documents—even if your retrieval step found 20 relevant ones.

Managing the Constraint

You must either:

  1. Retrieve fewer documents (top-3 instead of top-10)
  2. Truncate retrieved content (first 300 tokens per document)
  3. Choose models with larger windows (trade-off: cost and latency)
  4. Chunk more aggressively at index time (smaller, focused chunks)

Key Takeaway: Context window limits force you to be selective about which retrieved content makes it into your generation prompt—retrieval quality matters even more because you can't include everything.