Reference-Based vs Reference-Free Judging
What you'll learn: How to choose between comparing LLM outputs to ground truth references versus evaluating them standalone, and how to design prompts for each approach.
Core Concept
When using LLM-as-judge evaluation, you face a fundamental choice: should the judge compare model outputs against "gold standard" reference answers (reference-based), or evaluate them independently based solely on quality criteria (reference-free)?
Reference-based judging works like grading with an answer key. You provide the judge LLM with both the model's output and a human-written reference answer. The judge assesses similarity, correctness, or alignment. This works well for tasks with clear correct answers (factual QA, summarization, translation).
Reference-free judging evaluates outputs in isolation, like assessing creative writing without a model essay. The judge applies criteria like coherence, helpfulness, safety, or instruction-following without seeing any "correct" answer. This suits open-ended generation, creative tasks, or when multiple valid responses exist.
Prompt Design Differences
Reference-based prompts typically include:
- The input/question
- The reference answer
- The model output
- Instructions to compare them
Reference-free prompts contain:
- The input/question
- The model output only
- Evaluation rubric or criteria
- No ground truth to anchor judgment
Appropriate Use Cases
Use reference-based when:
- Objective correctness matters (math, facts)
- You have high-quality references available
- Tasks have relatively narrow answer spaces
Use reference-free when:
- Multiple valid responses exist (creative writing, dialogue)
- References don't exist or are costly to create
- Evaluating style, tone, or subjective quality
- Testing instruction-following beyond specific content
Key Takeaway: Reference-based judging anchors evaluation to ground truth and suits objective tasks, while reference-free judging assesses quality independently and works for open-ended generation—choose based on whether your task has clear correct answers.