Course contentsShow
← AI Engineering
Lesson 1344 of 1,88632. Fine-Tuning for ProductionFree lesson

Statistical Significance and Test Duration

Calculate required sample sizes and run tests long enough to detect meaningful differences between models.

Statistical Significance and Test Duration

You'll learn how to determine if your A/B test results are real improvements or just random noise, and how long to run tests for reliable conclusions.

The Core Challenge

After deploying two model variants in an A/B test, you need to answer: "Is Model B actually better, or did I just get lucky?" Running tests too short risks false positives—declaring a winner when none exists. Running them too long wastes resources and delays rollout.

Statistical significance tells you the probability that observed differences aren't due to chance. The industry standard is p < 0.05, meaning less than 5% chance your results are random.

Sample Size Calculation

Before launching, calculate the minimum sample size needed to detect a meaningful difference. This depends on:

  • Baseline metric (e.g., current 80% success rate)
  • Minimum detectable effect (e.g., want to catch 2% improvement)
  • Statistical power (typically 80%—ability to detect real differences)
  • Significance level (typically 0.05)

Think of it like weighing packages: detecting a 10-gram difference needs a more sensitive scale than detecting 1 kilogram. Smaller differences require larger samples.

Test Duration

Run tests until both conditions are met:

  1. You've reached minimum sample size
  2. You've captured typical usage patterns (weekdays, weekends, time zones)

Don't stop early just because results look good—this introduces peeking bias. A model might appear better on Monday but worse averaged over a full week.

For low-traffic systems, duration matters more than raw sample count. Even with enough requests, you need temporal coverage.

Key Takeaway: Calculate required samples upfront based on the smallest improvement worth detecting, then run tests through complete usage cycles before declaring statistical significance—rushing leads to false conclusions that waste deployment effort.