QA Evaluation Metrics
What you'll learn: How to measure the quality of answers produced by question answering systems using standardized metrics.
Why Metrics Matter
Just as you wouldn't grade an essay without criteria, we need objective ways to evaluate QA system performance. Different QA types require different metrics because extractive and abstractive answers have fundamentally different characteristics.
Extractive QA Metrics
For extractive QA (where answers are exact spans from context), we use two primary metrics:
Exact Match (EM)
This is the strictest measure: does the predicted answer exactly match the ground truth? No partial credit. If the correct answer is "Abraham Lincoln" but your model predicts "Lincoln," EM scores it as 0. Think of it like a true/false test.
F1 Score
F1 provides a gentler evaluation by measuring word overlap between prediction and ground truth. It treats the answer as a bag of words and calculates precision (how many predicted words are correct) and recall (how many ground-truth words were captured). The F1 score is their harmonic mean. If the answer is "Abraham Lincoln" and you predict "Lincoln," you'd get partial credit.
Abstractive QA Metrics
When systems generate free-form answers (abstractive QA), we need metrics that handle paraphrasing:
ROUGE
Measures n-gram overlap between generated and reference answers, focusing on recall. It's particularly useful for summarization-style answers.
BLEU
Originally from machine translation, BLEU emphasizes precision of n-gram matches. It penalizes answers that are too different from references.
Key Takeaway: Extractive QA uses Exact Match and F1 scores to measure span accuracy, while abstractive QA relies on ROUGE and BLEU to evaluate generated text quality through n-gram overlap.