Course contentsShow
← AI Engineering
Lesson 851 of 1,88621. Human Evaluation and FeedbackFree lesson

Comparison Data Collection Methods

Techniques for gathering pairwise comparisons: ranking outputs, preference selection, and comparative annotation interfaces.

Comparison Data Collection Methods

What you'll learn: You'll master the practical techniques for collecting pairwise comparison data that powers RLHF reward models.

Why Comparisons Matter

Now that you understand RLHF's three stages, you need quality comparison data to train your reward model. Instead of asking humans to rate outputs on a scale (which can be subjective and inconsistent), you present two model outputs side-by-side and ask: "Which is better?"

This comparative approach is more reliable—humans are better at choosing between options than assigning absolute scores.

Three Core Collection Methods

1. Ranking Outputs

Present multiple model responses (typically 2-4) for the same prompt and ask annotators to rank them from best to worst. This generates multiple pairwise preferences from a single annotation task.

Example: For "Explain photosynthesis," you show Response A, B, and C. If ranked C > A > B, you extract three comparisons: C>A, C>B, and A>B.

2. Preference Selection

The simplest approach: show two outputs and ask "Which is better?" or offer "A is better," "B is better," or "About equal." This binary choice is fast and intuitive for annotators.

3. Comparative Annotation Interfaces

Purpose-built UIs that highlight differences between outputs, often with side-by-side display, synchronized scrolling, and aspect-specific comparison (helpfulness, safety, accuracy). These tools reduce cognitive load and increase annotation quality.

Design Considerations

  • Randomize position (left/right) to avoid position bias
  • Include ties/equal options when genuinely similar
  • Collect rationales ("Why did you choose A?") to validate reasoning
  • Use stratified sampling to cover edge cases and diverse prompt types

Key Takeaway: Pairwise comparisons are the foundation of RLHF data collection—use ranking for efficiency, preference selection for simplicity, and specialized interfaces for quality at scale.