Course contentsShow
← Machine Learning and Deep Learning
Lesson 2022 of 3,53843. Retrieval-Augmented Generation: Advanced TechniquesFree lesson

Evaluating Query Rewriting Effectiveness

Metrics and methods for assessing whether rewritten queries improve retrieval quality and answer accuracy.

Evaluating Query Rewriting Effectiveness

What you'll learn: How to measure whether your query transformations actually improve retrieval quality and final answer accuracy in RAG systems.

Why Evaluation Matters

Query rewriting techniques like HyDE, decomposition, and multi-query generation promise better retrieval—but how do you know they're working? Without proper metrics, you're flying blind. Think of it like A/B testing different search terms: you need objective measures to prove one approach beats another.

Core Evaluation Dimensions

Retrieval Quality Metrics

Recall@K: Does the rewritten query retrieve more relevant documents in the top-K results? Compare the percentage of ground-truth relevant docs retrieved before and after rewriting.

MRR (Mean Reciprocal Rank): How quickly do relevant documents appear? A rewritten query that surfaces the best document first (rank 1) scores 1.0; at rank 2, it scores 0.5.

Precision@K: What fraction of retrieved documents are actually relevant? A rewritten query that retrieves 8 relevant docs out of 10 (vs. 5 out of 10 originally) shows clear improvement.

End-to-End Metrics

Answer Accuracy: Does the LLM produce correct answers more often with rewritten queries? Use human evaluation or automated fact-checking against labeled datasets.

Latency Impact: Query rewriting (especially LLM-based reformulation) adds overhead. Measure whether the quality gain justifies 100-500ms extra latency.

Cost Analysis: Multi-query generation might retrieve better context but also triples embedding and search costs. Track cost-per-query alongside quality metrics.

Comparative Testing

Run baseline queries alongside rewritten versions on the same document corpus. Use paired statistical tests to confirm improvements aren't random. Test across diverse question types—what helps factual queries may hurt conversational ones.

Key Takeaway: Effective query rewriting evaluation requires measuring both retrieval metrics (Recall@K, MRR, Precision@K) and downstream answer quality, while monitoring latency and cost trade-offs to ensure your transformations deliver real value.