Course contentsShow
← AI Engineering
Lesson 342 of 1,8869. Retrieval-Augmented Generation (RAG) - BasicsFree lesson

Chunk Size Trade-offs

Balancing precision vs context: small chunks for specificity, large chunks for broader understanding.

Chunk Size Trade-offs

What You'll Learn

You'll learn how to balance chunk size between precision and context when building RAG systems.

The Core Trade-off

When splitting documents for RAG, chunk size is a critical decision. Small chunks (50-200 tokens) provide precise, focused matches—your search returns exactly the sentence or paragraph that answers the query. Large chunks (500-1000+ tokens) provide broader context—more background information, but potentially dilute the relevance signal.

Think of it like choosing between a magnifying glass and a wide-angle lens. The magnifying glass (small chunks) shows fine detail but limited surroundings. The wide-angle lens (large chunks) captures the bigger picture but loses sharpness on any single element.

The Practical Balance

Small chunks excel when:

  • You need exact facts or definitions
  • Queries are specific ("What is the boiling point of water?")
  • Your embedding model works well with short text

Large chunks excel when:

  • Understanding requires surrounding context
  • Questions need multiple related ideas
  • You have limited context window space (fewer, richer chunks)

The problem: Small chunks might miss context that makes them understandable. Large chunks might score poorly on relevance because irrelevant text dilutes the embedding.

Finding Your Sweet Spot

Most production RAG systems use medium-sized chunks (200-500 tokens) with overlap. This balances specificity and context. You'll often need to experiment with your specific documents and query patterns—there's no universal answer.

Monitor retrieval quality: if answers feel incomplete, try larger chunks. If results seem off-topic, try smaller chunks.

Key Takeaway: Chunk size is a precision-vs-context trade-off; smaller chunks give specificity, larger chunks provide understanding, and most systems find success in the middle with experimentation.