Evaluating Query Rewriting Effectiveness
What you'll learn: How to measure whether your query transformations actually improve retrieval quality and final answer accuracy in RAG systems.
Why Evaluation Matters
Query rewriting techniques like HyDE, decomposition, and multi-query generation promise better retrieval—but how do you know they're working? Without proper metrics, you're flying blind. Think of it like A/B testing different search terms: you need objective measures to prove one approach beats another.
Core Evaluation Dimensions
Retrieval Quality Metrics
Recall@K: Does the rewritten query retrieve more relevant documents in the top-K results? Compare the percentage of ground-truth relevant docs retrieved before and after rewriting.
MRR (Mean Reciprocal Rank): How quickly do relevant documents appear? A rewritten query that surfaces the best document first (rank 1) scores 1.0; at rank 2, it scores 0.5.
Precision@K: What fraction of retrieved documents are actually relevant? A rewritten query that retrieves 8 relevant docs out of 10 (vs. 5 out of 10 originally) shows clear improvement.
End-to-End Metrics
Answer Accuracy: Does the LLM produce correct answers more often with rewritten queries? Use human evaluation or automated fact-checking against labeled datasets.
Latency Impact: Query rewriting (especially LLM-based reformulation) adds overhead. Measure whether the quality gain justifies 100-500ms extra latency.
Cost Analysis: Multi-query generation might retrieve better context but also triples embedding and search costs. Track cost-per-query alongside quality metrics.
Comparative Testing
Run baseline queries alongside rewritten versions on the same document corpus. Use paired statistical tests to confirm improvements aren't random. Test across diverse question types—what helps factual queries may hurt conversational ones.
Key Takeaway: Effective query rewriting evaluation requires measuring both retrieval metrics (Recall@K, MRR, Precision@K) and downstream answer quality, while monitoring latency and cost trade-offs to ensure your transformations deliver real value.