Course contentsShow
← AI Engineering
Lesson 343 of 1,8869. Retrieval-Augmented Generation (RAG) - BasicsFree lesson

Token Count Considerations

Managing chunk sizes relative to model context windows and ensuring retrieval fits generation limits.

Token Count Considerations

What you'll learn: How to balance chunk sizes with model token limits so your retrieval fits smoothly into generation without breaking or wasting context.


The Token Budget Challenge

When you retrieve chunks for RAG, they must fit inside your LLM's context window alongside your prompt and expected output. Think of it like packing a suitcase: you need room for the question, the retrieved knowledge, instructions, and the answer—all within a fixed capacity measured in tokens (not characters).

Why Token Counting Matters

A 512-character chunk isn't always 512 tokens. Different tokenizers break text differently—special characters, code, or non-English text often use more tokens than plain English. If you assume "1 character ≈ 1 token," you might:

  • Overflow the context window, causing the LLM to truncate your retrieval or reject the request
  • Waste precious context space by retrieving too little, leaving room unused
  • Cut off mid-sentence, confusing the model with incomplete information

Planning Your Token Budget

Before chunking, calculate:

  1. Model's total context window (e.g., 4096 tokens)
  2. Tokens for system prompt + user query (~100–300 tokens)
  3. Tokens reserved for generation (~500–1000 tokens)
  4. Remaining budget for retrieved chunks (the rest)

If you retrieve 5 chunks, each must average remaining_budget / 5 tokens. Use your embedding model's tokenizer to count actual tokens during chunking, not just characters.

The Retrieval Sweet Spot

Retrieve enough chunks to provide context, but not so many that you exceed limits or dilute relevance. Test with real queries and measure token usage before production.

Key Takeaway: Always count tokens—not characters—when sizing chunks, and leave headroom in your context window for prompts, retrieval, and generation to coexist comfortably.