Measuring Reranking Impact
What you'll learn: How to quantify whether your reranking and compression strategies actually improve retrieval quality using standard metrics.
Why Measurement Matters
You've implemented reranking with cross-encoders, tried compression, and tackled the lost-in-the-middle problem—but how do you know it's working? Measuring impact transforms guesswork into data-driven decisions about which retrieval optimizations to keep.
Core Metrics for Reranking
MRR (Mean Reciprocal Rank) measures how quickly users find the first relevant result. If the best answer appears at position 2, its reciprocal rank is 1/2 = 0.5. Average this across queries. Reranking should push relevant documents higher, increasing MRR.
NDCG (Normalized Discounted Cumulative Gain) accounts for both relevance scores and position. It rewards highly relevant results at the top more than those buried lower. NDCG ranges from 0 to 1; closer to 1 means better ranking quality. This metric is ideal when you have graded relevance labels (not just binary relevant/irrelevant).
End-to-End Accuracy measures what matters most: does the generated answer actually improve? Track metrics like answer correctness, citation accuracy, and user satisfaction. A reranker might boost MRR but hurt generation if it strips important context during compression.
The Evaluation Pattern
Compare "before reranking" vs "after reranking" on a test set of queries with known relevant documents. Calculate MRR and NDCG for both pipelines. Then evaluate the final generated answers—sometimes better retrieval ranking doesn't translate to better generation due to context window effects or compression artifacts.
Key Takeaway: Use MRR and NDCG to measure retrieval ranking quality, but always validate end-to-end generation accuracy—optimizing one component doesn't guarantee overall system improvement.