Diversity and MMR in Search Results
What You'll Learn
You'll learn how to use Maximal Marginal Relevance (MMR) to prevent your search results from being too repetitive, balancing similarity with variety.
The Problem with Pure Similarity Search
When you retrieve the top-K most similar vectors, you often get results that are too similar to each other. Imagine searching for "python programming" and getting back 10 tutorial articles that all say nearly the same thing—technically relevant, but not very useful.
Think of it like this: If you ask for restaurant recommendations and get five Italian restaurants all on the same street, you're missing out on diversity even though they're all "relevant" to food.
What is MMR?
Maximal Marginal Relevance is a re-ranking technique that balances two competing goals:
- Relevance: How similar is this result to your query?
- Diversity: How different is this result from what you've already selected?
MMR picks results iteratively. First, it chooses the most relevant item. Then, for each subsequent choice, it looks for items that are:
- Still reasonably relevant to your query
- Different from what's already been selected
The Trade-off Parameter
MMR uses a parameter (commonly λ, lambda) between 0 and 1:
- λ = 1: Pure relevance (ignore diversity)
- λ = 0: Maximum diversity (ignore relevance)
- λ = 0.5: Balanced approach
This lets you tune how much you value variety versus strict similarity.
Why This Matters
In RAG systems and search applications, diverse results give users broader context and reduce redundancy. Instead of five nearly-identical documents, you get five perspectives on the same topic.
Key Takeaway: MMR helps you avoid the "echo chamber" effect in vector search by iteratively selecting results that are relevant to the query but dissimilar to each other, controlled by a single diversity parameter.