Binary vs Scalar Judgments
What you'll learn: How to choose the right evaluation format—yes/no, scales, or numeric scores—when using LLM-as-a-Judge.
The Core Decision
When designing evaluation prompts for LLM-as-a-Judge, you need to decide how the judge model should report its assessment. Should it give a simple thumbs-up/thumbs-down? A 1-5 rating? A 0-100 score? This choice dramatically affects both the usefulness and reliability of your evaluations.
Three Common Formats
Binary judgments are yes/no or pass/fail decisions. They're perfect for clear-cut criteria: "Does the response contain personal information?" or "Did the model follow the instruction?"
Likert scales use discrete points (typically 1-5 or 1-7). These capture nuance while remaining structured: "Rate coherence from 1 (incoherent) to 5 (perfectly coherent)."
Numeric scores are continuous values, often 0-100. They provide fine-grained distinctions but can be harder for LLMs to apply consistently.
Choosing the Right Format
Use binary when the criterion is objective and unambiguous—either the model included a citation or it didn't.
Use scales when you need nuance but want consistency. A 5-point scale for "helpfulness" gives you enough gradation without overwhelming the judge model.
Use numeric scores sparingly, mainly when you need to aggregate many dimensions into a composite score or when small differences matter (like ranking many responses).
Think of it like grading: multiple choice (binary) is easier to score consistently than essay scoring (scalar), but essays capture more information.
Key Takeaway: Match your judgment format to your task needs—binary for clear criteria, scales for nuanced assessment, and numeric scores only when fine-grained discrimination is necessary and consistency can be maintained.