Human Feedback Collection for RLHF
What You'll Learn
You'll learn the practical methods for gathering preference data from human annotators to train reward models in RLHF pipelines.
Why We Collect Human Feedback
After instruction tuning, models can follow instructions but may still produce outputs that are unhelpful, harmful, or misaligned with human values. RLHF needs labeled data showing which outputs humans prefer—but humans aren't great at writing down explicit rules. Instead, we ask them to compare and judge outputs.
Three Core Collection Methods
1. Pairwise Comparisons
Present two model outputs for the same prompt. Annotators choose which one is better (or mark "tie"). This is the most common approach because:
- It's easier than absolute scoring
- Reduces annotator bias
- Mirrors natural human judgment
Example: For "Explain quantum computing," annotators compare Output A vs Output B and pick the clearer, more accurate response.
2. Rankings
Show multiple outputs (3-5 typically) and ask annotators to rank them from best to worst. This provides richer signal than pairwise comparisons but requires more cognitive effort.
3. Quality Ratings
Annotators assign absolute scores (e.g., 1-5 stars) to individual outputs based on criteria like helpfulness, safety, and truthfulness. Less commonly used alone because annotators often have inconsistent rating standards.
The Annotation Process
Think of annotators as teachers grading essays—they look for accuracy, helpfulness, harmlessness, and coherence. Their collective preferences become training data for the reward model, which learns to predict what humans will prefer.
High-quality data requires clear guidelines, annotator training, and often multiple ratings per example to ensure consistency.
Key Takeaway: Human feedback collection translates subjective human preferences into structured data through pairwise comparisons, rankings, or ratings—the foundation for training reward models that guide LLM alignment.