Course contentsShow
← Machine Learning and Deep Learning
Lesson 1318 of 3,53829. Core NLP TasksFree lesson

Translation Quality and Evaluation Metrics

Learn BLEU, METEOR, and chrF metrics for translation evaluation, understanding their strengths and limitations in measuring quality.

Translation Quality and Evaluation Metrics

You'll learn how to measure machine translation quality using BLEU, METEOR, and chrF metrics, and understand what each captures—and misses.

The Challenge of Evaluating Translation

When your transformer model translates text, how do you know if it did a good job? Unlike classification (right or wrong), translation has many valid outputs. "The cat sat on the mat" and "The feline rested on the rug" might both be correct translations of the same source sentence. We need automated metrics to evaluate quality at scale.

Three Key Metrics

BLEU (Bilingual Evaluation Understudy)

BLEU counts matching n-grams (word sequences) between the model's output and one or more human reference translations. It checks 1-grams (individual words), 2-grams (word pairs), 3-grams, and 4-grams, then combines these scores.

Strength: Fast, correlates reasonably with human judgment for corpus-level evaluation.
Limitation: Doesn't handle synonyms well ("happy" vs "joyful" scores zero similarity) and can penalize valid paraphrases.

METEOR (Metric for Evaluation of Translation with Explicit ORdering)

METEOR improves on BLEU by allowing stem matches ("running" ≈ "runs"), synonyms, and considering word order through alignment. It balances precision and recall.

Strength: Better correlation with human judgment at sentence level; handles linguistic variation.
Limitation: More computationally expensive; requires language-specific resources.

chrF (Character n-gram F-score)

chrF measures character-level overlap instead of words, calculating F-score from character n-grams.

Strength: Language-agnostic, handles morphologically rich languages better, robust to tokenization issues.
Limitation: May reward superficial similarity without capturing meaning.

Key Takeaway: BLEU, METEOR, and chrF each capture different aspects of translation quality—use multiple metrics together, and remember that human evaluation remains the gold standard for nuanced quality assessment.