Semantic Chunking
What you'll learn: How to use embeddings to create chunks that respect the natural semantic boundaries in your text, grouping related sentences together.
The Core Idea
Up until now, you've seen chunking strategies based on mechanical rules: fixed character counts, sentence breaks, or paragraph boundaries. Semantic chunking takes a smarter approach—it uses embeddings to measure the meaning of sentences and groups them based on semantic similarity.
Think of it like organizing a messy desk. Fixed-size chunking is like dividing items into equal piles regardless of what they are. Semantic chunking is like grouping pens together, papers together, and cables together—respecting what naturally belongs with what.
How It Works
- Embed each sentence individually using your embedding model
- Calculate similarity between consecutive sentences (cosine similarity between their embeddings)
- Identify semantic boundaries where similarity drops significantly—these mark topic shifts
- Group sentences between boundaries into coherent chunks
For example, if sentence 5 and 6 have low similarity (0.4) but sentences 6-9 are all highly similar (>0.8), you'd split before sentence 6 and keep 6-9 together.
Why This Matters for RAG
When your retrieval system fetches a chunk, you want that chunk to contain a complete, coherent idea—not half of one topic and half of another. Semantic chunking ensures each chunk is semantically self-contained, improving the quality of context passed to your LLM.
The trade-off? It's computationally more expensive than simple sentence-based chunking since you must embed every sentence and compute pairwise similarities.
Key Takeaway: Semantic chunking uses embedding similarity to detect natural topic boundaries, creating chunks that preserve semantic coherence rather than arbitrary text divisions.