Designing Comparison Tasks
You'll learn: How to create clear, consistent comparison tasks that help annotators accurately rank model outputs for RLHF training.
When collecting comparison data for RLHF, you need annotators to choose which of two (or more) model responses is better. But "better" is vague—annotators need crystal-clear instructions, criteria, and examples to make consistent decisions.
The Core Components
A well-designed comparison task has three elements:
1. Clear Criteria: Define exactly what "better" means for your use case. Is it factual accuracy? Helpfulness? Tone? Safety? List them in priority order. For a customer service bot, you might specify: "Choose the response that (1) answers the question completely, (2) maintains a friendly tone, and (3) is concise."
2. Detailed Instructions: Write step-by-step guidance for annotators. Include what to do when responses seem equally good (tie), when both are terrible, or when criteria conflict. Example: "If Response A is more accurate but Response B is friendlier, choose A—accuracy takes precedence."
3. Representative Examples: Provide annotated examples showing clear wins, edge cases, and ties. These calibrate annotators' judgment and reveal ambiguities in your criteria.
Think Like a Recipe Writer
Imagine explaining your task to someone who's never seen your product. You wouldn't say "make it taste good"—you'd specify ingredients, measurements, and techniques. Same here: replace subjective terms with observable behaviors. Instead of "more helpful," try "provides actionable next steps the user can follow."
Test your task design with a small pilot group. High inter-annotator agreement means your criteria work; disagreement signals you need clearer guidelines.
Key Takeaway: Effective comparison tasks transform vague preferences into concrete, teachable criteria with explicit instructions and calibration examples—the foundation of quality RLHF data.