Course contentsShow
← Machine Learning and Deep Learning
Lesson 1764 of 3,53838. Instruction Tuning and AlignmentFree lesson

Human Feedback Collection for RLHF

Methods for gathering preference data: pairwise comparisons, rankings, and quality ratings from human annotators.

Human Feedback Collection for RLHF

What You'll Learn

You'll learn the practical methods for gathering preference data from human annotators to train reward models in RLHF pipelines.

Why We Collect Human Feedback

After instruction tuning, models can follow instructions but may still produce outputs that are unhelpful, harmful, or misaligned with human values. RLHF needs labeled data showing which outputs humans prefer—but humans aren't great at writing down explicit rules. Instead, we ask them to compare and judge outputs.

Three Core Collection Methods

1. Pairwise Comparisons

Present two model outputs for the same prompt. Annotators choose which one is better (or mark "tie"). This is the most common approach because:

  • It's easier than absolute scoring
  • Reduces annotator bias
  • Mirrors natural human judgment

Example: For "Explain quantum computing," annotators compare Output A vs Output B and pick the clearer, more accurate response.

2. Rankings

Show multiple outputs (3-5 typically) and ask annotators to rank them from best to worst. This provides richer signal than pairwise comparisons but requires more cognitive effort.

3. Quality Ratings

Annotators assign absolute scores (e.g., 1-5 stars) to individual outputs based on criteria like helpfulness, safety, and truthfulness. Less commonly used alone because annotators often have inconsistent rating standards.

The Annotation Process

Think of annotators as teachers grading essays—they look for accuracy, helpfulness, harmlessness, and coherence. Their collective preferences become training data for the reward model, which learns to predict what humans will prefer.

High-quality data requires clear guidelines, annotator training, and often multiple ratings per example to ensure consistency.

Key Takeaway: Human feedback collection translates subjective human preferences into structured data through pairwise comparisons, rankings, or ratings—the foundation for training reward models that guide LLM alignment.