Course contentsShow
← Machine Learning and Deep Learning
Lesson 1324 of 3,53829. Core NLP TasksFree lesson

Evaluation Challenges in Generation

Learn why generation evaluation is difficult and explore automatic metrics, human evaluation protocols, and their trade-offs.

Evaluation Challenges in Generation

What you'll learn: Why measuring the quality of generated text is surprisingly difficult and what tools exist to tackle this challenge.

The Core Problem

When your model generates text—summaries, translations, or responses—how do you know if it's good? Unlike classification tasks where you have clear right/wrong answers, generation has multiple valid outputs. A sentence can be rephrased dozens of ways while preserving meaning, making evaluation far more nuanced.

Why It's Hard

Imagine asking two people to summarize the same article. Both summaries might be excellent yet completely different in wording, length, and focus. Your model faces the same challenge: there's no single "correct" generation to compare against.

Human judgment is the gold standard—people can assess fluency, coherence, factuality, and relevance—but it's slow, expensive, and sometimes inconsistent between reviewers.

Automatic Metrics

To scale evaluation, we use automatic metrics like BLEU (for translation) and ROUGE (for summarization). These compare generated text to reference texts using n-gram overlap or other statistical measures. They're fast and reproducible but have serious limitations:

  • They don't understand meaning—just surface-level word matches
  • They penalize valid paraphrases
  • They correlate poorly with human judgment on creative tasks

Human Evaluation Protocols

Human evaluation asks reviewers to rate dimensions like fluency, relevance, or factuality on scales (1-5). The trade-off: it's accurate but requires multiple annotators per example, inter-rater agreement checks, and substantial time/cost.

The Trade-off

You'll typically use automatic metrics for rapid iteration during development, then validate with human evaluation before deployment. Neither approach is perfect—generation evaluation remains an open research problem.

Key Takeaway: Generation evaluation requires balancing fast automatic metrics (limited accuracy) with expensive human evaluation (high accuracy), because multiple valid outputs make objective measurement inherently difficult.