Course contentsShow
← AI Engineering
Lesson 871 of 1,88621. Human Evaluation and FeedbackFree lesson

Statistical Power and Sample Size for AI Tests

Calculate required sample sizes accounting for variance in AI outputs and realistic effect sizes.

Statistical Power and Sample Size for AI Tests

You'll learn how to calculate the minimum number of users or samples needed to detect real improvements in your AI system, accounting for the inherent randomness in LLM outputs.

Why Sample Size Matters for AI Tests

When A/B testing two prompts or models, you need enough data to distinguish real improvements from random noise. Statistical power is your probability of detecting a true difference when it exists—typically set at 80% or higher.

Unlike traditional software where feature X either works or doesn't, AI outputs vary naturally. The same prompt can produce slightly different responses. This variance means you need larger samples to confidently say "Model B is actually better than Model A."

Calculating Required Sample Size

The formula depends on four factors:

  1. Significance level (α): Usually 0.05 (5% false positive rate)
  2. Statistical power (1-β): Usually 0.80 (80% detection rate)
  3. Effect size: The minimum improvement you care about (e.g., 5% higher success rate)
  4. Variance: How much your AI's outputs naturally fluctuate

The smaller the effect size you want to detect, the more samples you need. If Model B is only 2% better than Model A, you'll need thousands of evaluations. If it's 20% better, hundreds might suffice.

For AI systems with high output variance (like creative generation tasks), multiply your traditional sample size estimate by 1.5–3×. If your metric is success rate, use standard proportion tests. For scalar scores (1-10 ratings), use t-test calculators with realistic standard deviation estimates from pilot data.

Key Takeaway: Always run a small pilot test first to estimate variance, then calculate required sample size based on the smallest improvement that matters to your users—don't guess and risk underpowered tests that waste time.