Course contentsShow
← AI Engineering
Lesson 430 of 1,88611. RAG - Improving GenerationFree lesson

Diversity-Aware Selection

Select diverse chunks to avoid redundancy and provide broader context to the generation model.

Diversity-Aware Selection

What you'll learn: How to select diverse chunks from your retrieval results to avoid redundancy and give the generation model richer context.

The Problem with Pure Similarity Ranking

You've learned Top-K selection strategies that return the most similar chunks to a query. But here's the catch: your top 5 results might all say nearly the same thing! If someone searches "What is photosynthesis?" and all five retrieved chunks are from the same textbook chapter, you're wasting precious context window space on repetitive information.

What Is Diversity-Aware Selection?

Diversity-aware selection balances relevance with uniqueness. Instead of only taking the top-scoring chunks, you intentionally select chunks that cover different aspects, perspectives, or subtopics of the query. Think of it like building a well-rounded meal—you wouldn't serve five different pasta dishes; you'd want protein, vegetables, and sides.

Why It Matters

When your generation model receives diverse context:

  • It can synthesize more comprehensive answers
  • Users get broader coverage of their question
  • You reduce the "echo chamber" effect of similar sources
  • Better answers emerge from complementary information rather than repetition

Common Approaches

After applying relevance filtering and reranking, diversity selection might:

  • Cluster similar chunks and pick one representative from each cluster
  • Use maximal marginal relevance (MMR) to penalize similarity to already-selected chunks
  • Compare embeddings between candidates and select those with distinct semantic content
  • Use metadata (different documents, authors, sections) as diversity signals

Key Takeaway: Diversity-aware selection ensures your retrieved context provides complementary information rather than redundant repetition, maximizing the value of every chunk you pass to the generation model.