Diversity in Reranking
What you'll learn: How to prevent redundant results by balancing relevance with diversity when reranking retrieved documents.
The Problem with Pure Relevance
When reranking retrieval results using cross-encoders, you typically sort documents purely by relevance score. But this creates a problem: your top-10 results might all say nearly the same thing. Imagine searching "Python web frameworks" and getting ten slightly different articles all about Django—highly relevant, but not helpful if the user wants to compare options.
Maximal Marginal Relevance (MMR)
MMR is a classic technique that balances relevance with diversity. Instead of greedily picking the most relevant documents, MMR iteratively selects documents that are:
- Relevant to the query
- Dissimilar to already-selected documents
The formula uses a tunable parameter λ (lambda) between 0 and 1:
MMR = λ × relevance_score - (1-λ) × max_similarity_to_selected
- λ = 1: pure relevance (ignores diversity)
- λ = 0: pure diversity (ignores relevance)
- λ = 0.7: common balanced choice
How It Works
Think of building a playlist: you don't just pick the ten most popular songs—you want variety in tempo, mood, and genre while still keeping quality high. MMR applies this same principle to document selection.
The algorithm works greedily: after selecting each document, it penalizes remaining candidates that are too similar to what's already chosen, measured using embedding similarity (cosine, dot product, etc.).
Beyond MMR
Other diversity techniques include clustering-based selection (pick representatives from different clusters) and determinantal point processes, though MMR remains popular for its simplicity and effectiveness.
Key Takeaway: MMR and diversity-aware reranking prevent result redundancy by trading off pure relevance for broader coverage, ensuring users see varied perspectives rather than repetitive information.