Course contentsShow
← AI Engineering
Lesson 350 of 1,8869. Retrieval-Augmented Generation (RAG) - BasicsFree lesson

Context Window Constraints

Working within token limits and understanding how context window size affects retrieval decisions.

Context Window Constraints

What you'll learn: How token limits restrict how much retrieved context you can feed to an LLM and why this shapes your retrieval strategy.


The Problem: LLMs Have Memory Limits

Every large language model has a context window — a hard limit on how many tokens (roughly words or word pieces) it can process in a single request. This includes your prompt, all retrieved documents, and space for the model's response.

Think of it like a suitcase: you can only pack so much. If you retrieve 50 document chunks but your context window only fits 10, you're forced to make tough choices about what to include.

Why This Matters for RAG

In the three-step RAG pipeline (retrieve → augment → generate), the augment step must fit retrieved context into the model's token budget. If your retrieval step returns too much content, you'll need to:

  • Trim or truncate chunks (losing information)
  • Retrieve fewer chunks (risking lower recall)
  • Use more selective metadata filtering
  • Choose smaller chunk sizes at index time

Common context windows range from 4K tokens (older models) to 128K+ tokens (modern models like GPT-4 Turbo). A 4K window might only accommodate 3-5 medium-length document chunks plus your prompt and response space.

Balancing Act

Your chunk size strategy from earlier lessons directly impacts how many chunks fit. Smaller chunks = more chunks possible, but each carries less context. Larger chunks = fewer fit, but each is more comprehensive.

You must also reserve tokens for:

  • System instructions
  • The user's query
  • The model's generated answer

Key Takeaway: Context window size is a hard constraint that forces you to carefully balance chunk size, number of retrieved results, and metadata filtering to maximize relevant information while staying within token limits.