Course contentsShow
← Machine Learning and Deep Learning
Lesson 3108 of 3,53868. Evaluation in PracticeFree lesson

Types of Human Evaluation Tasks

Survey of evaluation formats: pairwise comparisons, Likert scales, binary judgments, and free-form feedback.

Types of Human Evaluation Tasks

What you'll learn: The main formats used to collect human judgments when evaluating ML systems, from simple binary choices to open-ended feedback.

Why Different Formats Exist

When your ML model generates text, ranks recommendations, or creates images, automated metrics can only tell part of the story. Human evaluators provide the ground truth for qualities like helpfulness, coherence, or creativity. Different evaluation formats trade off between ease of collection, statistical power, and richness of insight.

Common Evaluation Formats

Pairwise Comparisons

Evaluators choose which of two model outputs is better. Like judging "Which summary is clearer: A or B?" This format is intuitive and reduces absolute judgment bias (no need to define "what is a 5/10?"), but requires more samples since you only compare two items at a time.

Likert Scales

Evaluators rate outputs on a numeric scale, typically 1-5 or 1-7 (e.g., "Rate this translation's accuracy: 1=Poor to 5=Excellent"). This provides granular data and works well for measuring intensity, but different annotators may interpret scales differently.

Binary Judgments

Simple yes/no or acceptable/unacceptable evaluations. "Is this response safe?" or "Does this image contain a cat?" Fast to collect and statistically straightforward, but loses nuance about degree or severity.

Free-Form Feedback

Evaluators write open-ended comments explaining issues or strengths. Think of this as qualitative research—you discover unexpected failure modes and edge cases, but analysis is time-consuming and doesn't easily convert to quantitative metrics.

Choosing Your Format

Match the format to your goal: pairwise for relative quality ranking, Likert for measuring dimensions, binary for pass/fail criteria, and free-form for exploration and root cause analysis.

Key Takeaway: Human evaluation formats—pairwise comparisons, Likert scales, binary judgments, and free-form feedback—each offer different trade-offs between collection speed, statistical power, and insight depth; choose based on what aspect of model quality you need to measure.