Course contentsShow
← Machine Learning and Deep Learning
Lesson 3444 of 3,53875. LLM Safety and Alignment ChallengesFree lesson

Human Label Quality and Inconsistency

The fundamental challenge of noisy, biased, and inconsistent human preference annotations in RLHF datasets.

Human Label Quality and Inconsistency

What you'll learn: Why imperfect human annotations create fundamental challenges for aligning language models through RLHF.

The Core Problem

In RLHF, we train reward models on human preference data — rankings or comparisons that tell us which AI responses are better. But humans aren't perfect judges. Different annotators disagree about what's "better," the same annotator makes different choices on different days, and systematic biases creep into labels. This noise pollutes the training signal your reward model learns from.

Why This Matters for Alignment

Imagine training a reward model where:

  • Annotator A prefers concise answers, Annotator B prefers detailed ones
  • The same response gets labeled "good" on Monday, "bad" on Friday
  • Cultural biases favor certain writing styles or perspectives
  • Complex ethical questions receive inconsistent ratings

Your reward model tries to learn patterns from this messy data. It can't distinguish genuine quality signals from human error or bias. The result? The model learns to exploit the noise rather than capture true human values — amplifying sycophancy, overfitting to arbitrary preferences, or learning spurious correlations.

The Measurement Challenge

Unlike supervised learning where labels are often objective ("this is a cat"), preference judgments are inherently subjective and context-dependent. When inter-annotator agreement is low, you're essentially asking the model to learn from contradictory training data. This is a data quality problem at the foundation of alignment.

Key Takeaway: Human preference labels are noisy, inconsistent, and biased — creating an unreliable foundation for reward models that directly impacts whether RLHF produces genuinely aligned behavior or simply overfits to human labeling quirks.