Course contentsShow
← Machine Learning and Deep Learning
Lesson 3168 of 3,53869. LLM EvaluationFree lesson

Reference-Based vs Reference-Free Judging

Evaluating with or without ground truth references, appropriate use cases for each, and prompt design differences.

Reference-Based vs Reference-Free Judging

What you'll learn: How to choose between comparing LLM outputs to ground truth references versus evaluating them standalone, and how to design prompts for each approach.

Core Concept

When using LLM-as-judge evaluation, you face a fundamental choice: should the judge compare model outputs against "gold standard" reference answers (reference-based), or evaluate them independently based solely on quality criteria (reference-free)?

Reference-based judging works like grading with an answer key. You provide the judge LLM with both the model's output and a human-written reference answer. The judge assesses similarity, correctness, or alignment. This works well for tasks with clear correct answers (factual QA, summarization, translation).

Reference-free judging evaluates outputs in isolation, like assessing creative writing without a model essay. The judge applies criteria like coherence, helpfulness, safety, or instruction-following without seeing any "correct" answer. This suits open-ended generation, creative tasks, or when multiple valid responses exist.

Prompt Design Differences

Reference-based prompts typically include:

  • The input/question
  • The reference answer
  • The model output
  • Instructions to compare them

Reference-free prompts contain:

  • The input/question
  • The model output only
  • Evaluation rubric or criteria
  • No ground truth to anchor judgment

Appropriate Use Cases

Use reference-based when:

  • Objective correctness matters (math, facts)
  • You have high-quality references available
  • Tasks have relatively narrow answer spaces

Use reference-free when:

  • Multiple valid responses exist (creative writing, dialogue)
  • References don't exist or are costly to create
  • Evaluating style, tone, or subjective quality
  • Testing instruction-following beyond specific content

Key Takeaway: Reference-based judging anchors evaluation to ground truth and suits objective tasks, while reference-free judging assesses quality independently and works for open-ended generation—choose based on whether your task has clear correct answers.