Annotation Cost and Sample Size
You'll learn how to determine the right number of human annotations while staying within budget and ensuring your evaluation results are statistically meaningful.
The Balancing Act
When planning human evaluation, you face a classic trade-off: more annotations give you more confidence in your results, but each annotation costs time and money. Hiring three annotators to rate 1,000 examples might cost thousands of dollars and take weeks—but evaluating only 10 examples won't tell you anything reliable about your system's real performance.
Understanding the Components
Cost per annotation includes:
- Annotator time (often $15-50/hour depending on expertise)
- Platform fees (annotation tools charge per task)
- Quality control overhead (reviewing edge cases, resolving disagreements)
- Training time for new annotators
Sample size determines:
- Statistical confidence (can you trust the numbers?)
- Coverage of edge cases (rare but important scenarios)
- Ability to detect meaningful differences between models
Finding the Sweet Spot
Start by estimating your minimum detectable effect: if Model A has 75% accuracy and Model B has 78%, do you care? If you need to detect a 3% difference with 95% confidence, you might need 300+ annotations. For a 10% difference, perhaps 100 is enough.
Use power analysis formulas (or online calculators) to determine sample size based on expected effect size and desired confidence level. For most LLM evaluations, 100-500 annotations per evaluation dimension strikes a practical balance.
Key Takeaway: Calculate your required sample size based on the smallest performance difference you care about, then work backward to budget—or accept lower confidence with fewer samples if resources are limited.