Position Bias in LLM Judges
What you'll learn: How the order in which outputs are presented to an LLM judge systematically affects its judgments, and practical strategies to measure and mitigate this bias.
The Problem: Order Matters More Than It Should
When using LLM-as-Judge for pairwise comparison, you present two model outputs (say, Output A and Output B) and ask the judge to pick the better one. Ideally, the judge's decision should depend only on quality. In practice, however, position bias means the judge favors whichever output appears first (or sometimes last), regardless of actual merit.
Think of it like a restaurant menu: items at the top get ordered more often, not because they're better, but because they're seen first. Similarly, an LLM judge might systematically prefer "Output A" simply because it appeared before "Output B" in the prompt.
Measuring Position Bias
To detect position bias, evaluate the same pair twice with swapped positions:
- First evaluation: "Output A vs Output B"
- Second evaluation: "Output B vs Output A"
If the judge is unbiased, it should give consistent verdicts. If it prefers A in the first trial but B in the second (when B is now first), you've detected position bias. Calculate the bias rate as the percentage of inconsistent judgments across many pairs.
Mitigation Strategies
Swapping is the gold standard: always evaluate each pair twice with reversed order, then aggregate results (e.g., count a win only if the model wins in both positions, or average the scores). This doubles evaluation cost but dramatically improves fairness.
You can also experiment with judge prompt design—explicitly instructing the model to "ignore position" or randomizing label names—though swapping remains most reliable.
Key Takeaway: Position bias causes LLM judges to favor outputs based on presentation order; measure it through position swapping and mitigate by evaluating each pair in both orders.