Generation Quality Metrics
What you'll learn: How to measure the quality of generated text by comparing it to reference outputs using BLEU, ROUGE, and METEOR scores.
Why We Need Generation Metrics
When your LLM summarizes articles, translates text, or generates descriptions, you need objective ways to measure quality. Unlike classification (right/wrong) or retrieval (relevant/irrelevant), generation tasks produce open-ended text. Generation quality metrics solve this by comparing your LLM's output against one or more reference "gold standard" texts.
The Big Three Metrics
BLEU (Bilingual Evaluation Understudy)
Originally designed for translation, BLEU measures n-gram overlap between generated and reference text. It checks how many word sequences (1-word, 2-word, 3-word, etc.) appear in both texts.
Think of it like checking how many puzzle pieces from your generated text fit into the reference text puzzle.
Best for: Tasks where exact wording matters (translation, structured summaries).
ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
ROUGE focuses on recall — how much of the reference text appears in the generation. ROUGE-N checks n-gram overlap, while ROUGE-L measures longest common subsequence.
Like asking: "Did the summary capture the key facts from the reference?"
Best for: Summarization tasks where covering important content matters more than exact phrasing.
METEOR (Metric for Evaluation of Translation with Explicit Ordering)
METEOR is more sophisticated — it considers synonyms, stemming (walk/walked), and word order. It balances precision and recall with a harmonic mean.
Like a teacher who gives credit for saying the same thing different ways.
Best for: Tasks requiring semantic similarity, not just exact matches.
When to Use Them
These metrics work when you have reference texts to compare against. They're automated, fast, and consistent — perfect for continuous evaluation pipelines tracking improvements over time.
Key Takeaway: BLEU, ROUGE, and METEOR let you quantify generation quality by comparing outputs to reference texts, each emphasizing different aspects like exact matches, content coverage, or semantic similarity.