Course contentsShow
← Machine Learning and Deep Learning
Lesson 1991 of 3,53842. Retrieval-Augmented Generation: FoundationsFree lesson

Chunk Size Trade-offs

Balancing chunk size between retrieval precision, context completeness, and embedding model limitations.

Chunk Size Trade-offs

What you'll learn: How to balance chunk size to optimize retrieval accuracy, context quality, and system constraints in RAG pipelines.

Choosing the right chunk size is one of RAG's most critical design decisions. It's like choosing the size of excerpts in a research paper's literature review: too small and you lose context; too large and you dilute relevance.

The Core Tension

Small chunks (e.g., 128–256 tokens):

  • Precision advantage: Each retrieved chunk closely matches the query semantically
  • Context loss: May cut off important surrounding information
  • More chunks needed: You might need to retrieve 10+ chunks to get complete answers
  • Embedding quality: Short spans may lack sufficient context for meaningful embeddings

Large chunks (e.g., 1024–2048 tokens):

  • Context completeness: Preserve narrative flow and relationships
  • Precision penalty: Retrieved chunks contain irrelevant information alongside the target content
  • Token cost: Fewer chunks needed, but each chunk consumes more of your LLM's context window
  • Embedding dilution: The embedding represents a broader semantic space, potentially reducing retrieval accuracy

The Sweet Spot

Most production systems converge on 256–512 tokens with 50–100 token overlap (from Overlapping Chunks). This balances:

  • Sufficient context for standalone meaning
  • Manageable LLM context consumption
  • Embedding model effectiveness (most models are trained on similar-length passages)
  • Retrieval precision

Practical Considerations

Your chunk size should respect:

  • Embedding model limits: Models like Sentence Transformers typically have 512-token maximums
  • Domain characteristics: Technical documentation may need larger chunks; FAQ-style content works with smaller
  • Query patterns: Complex questions benefit from larger context; factoid queries work with smaller chunks

Key Takeaway: Chunk size is a three-way trade-off between retrieval precision, context completeness, and system constraints—start with 256–512 tokens and tune based on your retrieval quality metrics and token budgets.