Course contentsShow
← Machine Learning and Deep Learning
Lesson 3114 of 3,53868. Evaluation in PracticeFree lesson

Aggregating Human Judgments

Combining multiple annotations with majority voting, weighted averages, and expert adjudication strategies.

Aggregating Human Judgments

What you'll learn: How to combine multiple human annotations into a single, reliable label using majority voting, weighted strategies, and expert adjudication.

The Core Challenge

When you collect human annotations (e.g., "Is this email spam?"), you often get multiple opinions per data point. How do you combine these into one ground truth label? Simply picking one at random wastes information; averaging everything equally ignores annotator quality differences.

Three Main Aggregation Strategies

1. Majority Voting

The simplest approach: the label chosen by most annotators wins. If 3 out of 5 annotators say "spam," that's your final label. Works well when annotators have similar expertise and the task is clear-cut.

Analogy: A jury verdict where everyone's vote counts equally.

2. Weighted Averaging

Not all annotators are equally reliable. Weight each vote by annotator quality metrics like historical accuracy, inter-annotator agreement scores, or expertise level. An expert's "spam" vote might count 2×, while a novice counts 0.5×.

This captures that your senior linguist's judgment on sentiment should outweigh a crowd worker who rushes through tasks.

3. Expert Adjudication

For disagreements or edge cases, route the example to a senior expert who makes the final call. Often combined with voting: use majority vote when agreement is high, escalate conflicts to experts.

Analogy: A court case where judges review what the jury couldn't decide.

Practical Considerations

  • Confidence thresholds: Only accept aggregated labels when agreement exceeds a threshold (e.g., 80% consensus)
  • Tie-breaking: Define clear rules when votes split evenly
  • Cost vs quality: Expert adjudication is expensive but accurate; majority voting is cheap but noisier

Key Takeaway: Aggregating human judgments requires balancing simplicity (majority voting), quality awareness (weighted averaging), and expert oversight (adjudication) to produce reliable ground truth labels from multiple noisy human inputs.