Rating Scales and Scoring Systems
What you'll learn: How to select the right rating format—Likert scales, binary ratings, or comparative judgments—to collect meaningful human feedback on LLM outputs.
Once you've built your evaluation rubric (as covered in the previous lesson), you need to decide how annotators will express their judgments. This choice directly impacts data quality, annotator efficiency, and how you'll aggregate results.
Three Common Rating Approaches
Binary ratings (yes/no, pass/fail) are fastest and simplest. Use them when quality is clearly distinguishable: "Does this summary contain factual errors?" or "Did the model follow the instruction?" They produce clean data but lose nuance—you can't distinguish "excellent" from "barely acceptable."
Likert scales (typically 1-5 or 1-7) capture gradations: "Rate fluency from 1 (incomprehensible) to 5 (perfectly natural)." They're ideal when quality exists on a spectrum. The sweet spot is usually 5 points—fewer feels limiting, more creates confusion about what each level means. Always anchor your scale with clear descriptions for at least the endpoints and midpoint.
Comparative judgments (pairwise or ranking) ask annotators to compare outputs: "Which response is more helpful, A or B?" This approach reduces subjective calibration issues—it's easier to say "A is better than B" than to assign absolute scores. Use this when evaluating model versions against each other or when absolute quality is hard to define.
Choosing Your Scale
Match the scale to your metric's nature. Factuality? Binary works. Helpfulness? Likert captures nuance. Model comparison? Pairwise judgments shine. Consider annotator fatigue too—binary ratings process fastest, while pairwise comparisons scale quadratically with options.
Key Takeaway: Select binary ratings for clear-cut decisions, Likert scales for nuanced quality dimensions, and comparative judgments when relative performance matters more than absolute scores—align your rating system with both the metric's nature and your evaluation goals.