Course contentsShow
← AI Engineering
Lesson 855 of 1,88621. Human Evaluation and FeedbackFree lesson

Handling Disagreement and Ambiguity

Strategies for resolving conflicting annotations, identifying subjective cases, and modeling uncertainty in preferences.

Handling Disagreement and Ambiguity

What you'll learn: How to resolve conflicting human annotations and model uncertainty when preferences aren't clear-cut.

When collecting comparison data for RLHF, you'll inevitably encounter situations where annotators disagree about which model output is better, or where the "right" answer genuinely depends on subjective judgment. This isn't a flaw in your process—it's valuable signal about the ambiguity inherent in human preferences.

Why Disagreement Happens

Annotators may disagree because:

  • The outputs are genuinely similar in quality
  • The task involves subjective preferences (tone, style, humor)
  • The rubric leaves room for interpretation
  • Different annotators weight criteria differently (accuracy vs. helpfulness)

Think of it like asking five friends to recommend the "best" restaurant—you'll get different answers based on their values, and all might be valid.

Strategies for Resolution

Majority voting: Accept the label chosen by most annotators, but flag low-agreement samples.

Confidence weighting: Track how strongly annotators feel (e.g., "slightly better" vs. "much better") and weight accordingly.

Tie handling: When it's 50/50, either exclude the pair or label it as "no preference"—both approaches teach your model something different.

Expert adjudication: Have senior annotators review high-disagreement cases to establish ground truth.

Modeling Uncertainty

Don't discard disagreement data—use it! Low-agreement samples indicate areas where your model should express uncertainty rather than overconfident preferences. You can:

  • Include "tie" labels in your training data
  • Weight training examples by annotator consensus
  • Build separate models for high-confidence vs. ambiguous cases

Key Takeaway: Disagreement isn't noise to eliminate—it's information about where human preferences are genuinely ambiguous, helping you build models that respect the subjective nature of many AI tasks.