Course contentsShow
← Machine Learning and Deep Learning
Lesson 3169 of 3,53869. LLM EvaluationFree lesson

Calibrating LLM Judges Against Human Ratings

Measuring agreement between LLM judgments and human preferences, computing correlation metrics, and validation strategies.

Calibrating LLM Judges Against Human Ratings

What you'll learn: How to validate and tune LLM judges by measuring how well their judgments align with human preferences.

Why Calibration Matters

An LLM judge is only useful if its ratings correlate with what humans actually prefer. Without calibration, you might be optimizing for judgments that don't reflect real-world quality. Think of it like calibrating a thermometer against known reference temperatures—you need ground truth to trust your instrument.

Core Calibration Process

1. Collect paired judgments: Gather the same evaluation tasks judged by both humans and your LLM. For example, 500 response pairs rated by both human annotators and GPT-4-as-judge.

2. Compute correlation metrics: Calculate how strongly the two sets of ratings agree:

  • Spearman's rank correlation for ordinal judgments (which is better?)
  • Pearson correlation for continuous scores (rating 1-10)
  • Cohen's kappa for categorical agreement (good/bad/neutral)
  • Accuracy when treating pairwise preferences as classification

3. Analyze disagreements: Identify systematic patterns where the LLM judge diverges—certain topics, response lengths, or writing styles where calibration fails.

Validation Strategies

  • Holdout validation: Reserve a subset of human-judged examples not used during prompt engineering
  • Stratified sampling: Ensure your calibration set covers diverse scenarios (multi-aspect evaluation dimensions, reference-based and reference-free contexts)
  • Iterative refinement: Adjust judge prompts or chain-of-thought reasoning based on correlation gaps, then re-measure
  • Cross-model validation: Test whether calibration holds when switching judge models

A correlation above 0.7-0.8 typically indicates reasonable alignment, though domain requirements vary.

Key Takeaway: Calibrating LLM judges means quantifying agreement with human ratings using correlation metrics, analyzing failure modes, and iteratively refining prompts until the judge reliably approximates human preferences on held-out evaluation samples.