Course contentsShow
← AI Engineering
Lesson 812 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Binary vs Scalar Judgments

Choosing between yes/no evaluations, Likert scales, or numeric scores based on task requirements.

Binary vs Scalar Judgments

What you'll learn: How to choose the right evaluation format—yes/no, scales, or numeric scores—when using LLM-as-a-Judge.

The Core Decision

When designing evaluation prompts for LLM-as-a-Judge, you need to decide how the judge model should report its assessment. Should it give a simple thumbs-up/thumbs-down? A 1-5 rating? A 0-100 score? This choice dramatically affects both the usefulness and reliability of your evaluations.

Three Common Formats

Binary judgments are yes/no or pass/fail decisions. They're perfect for clear-cut criteria: "Does the response contain personal information?" or "Did the model follow the instruction?"

Likert scales use discrete points (typically 1-5 or 1-7). These capture nuance while remaining structured: "Rate coherence from 1 (incoherent) to 5 (perfectly coherent)."

Numeric scores are continuous values, often 0-100. They provide fine-grained distinctions but can be harder for LLMs to apply consistently.

Choosing the Right Format

Use binary when the criterion is objective and unambiguous—either the model included a citation or it didn't.

Use scales when you need nuance but want consistency. A 5-point scale for "helpfulness" gives you enough gradation without overwhelming the judge model.

Use numeric scores sparingly, mainly when you need to aggregate many dimensions into a composite score or when small differences matter (like ranking many responses).

Think of it like grading: multiple choice (binary) is easier to score consistently than essay scoring (scalar), but essays capture more information.

Key Takeaway: Match your judgment format to your task needs—binary for clear criteria, scales for nuanced assessment, and numeric scores only when fine-grained discrimination is necessary and consistency can be maintained.