Course contentsShow
← AI Engineering
Lesson 847 of 1,88621. Human Evaluation and FeedbackFree lesson

Annotation Cost and Sample Size

Determining how many annotations you need and balancing budget constraints with statistical validity.

Annotation Cost and Sample Size

You'll learn how to determine the right number of human annotations while staying within budget and ensuring your evaluation results are statistically meaningful.

The Balancing Act

When planning human evaluation, you face a classic trade-off: more annotations give you more confidence in your results, but each annotation costs time and money. Hiring three annotators to rate 1,000 examples might cost thousands of dollars and take weeks—but evaluating only 10 examples won't tell you anything reliable about your system's real performance.

Understanding the Components

Cost per annotation includes:

  • Annotator time (often $15-50/hour depending on expertise)
  • Platform fees (annotation tools charge per task)
  • Quality control overhead (reviewing edge cases, resolving disagreements)
  • Training time for new annotators

Sample size determines:

  • Statistical confidence (can you trust the numbers?)
  • Coverage of edge cases (rare but important scenarios)
  • Ability to detect meaningful differences between models

Finding the Sweet Spot

Start by estimating your minimum detectable effect: if Model A has 75% accuracy and Model B has 78%, do you care? If you need to detect a 3% difference with 95% confidence, you might need 300+ annotations. For a 10% difference, perhaps 100 is enough.

Use power analysis formulas (or online calculators) to determine sample size based on expected effect size and desired confidence level. For most LLM evaluations, 100-500 annotations per evaluation dimension strikes a practical balance.

Key Takeaway: Calculate your required sample size based on the smallest performance difference you care about, then work backward to budget—or accept lower confidence with fewer samples if resources are limited.