Randomization and Sample Size Calculation
What You'll Learn
You'll learn how to properly randomize experiments and calculate the sample sizes needed to make statistically valid decisions about your AI features.
Why This Matters
When you run A/B tests (covered in lesson 1859), you need enough users in each group and proper random assignment to trust your results. Get this wrong, and you might ship a worse model or kill a better one based on noise rather than real performance differences.
Core Concepts
Randomization Strategies
Random assignment ensures each user has an equal chance of seeing variant A or B, preventing bias. Without it, you might accidentally send all mobile users to one variant and desktop users to another, confusing device effects with your AI feature's impact.
Common approaches:
- Hash-based: Use a hash function on user IDs to deterministically assign groups (consistent across sessions)
- True random: Generate random numbers for each decision (less reproducible)
- Stratified: Ensure equal representation across important segments (e.g., new vs. returning users)
Sample Size Calculation
You need enough data to detect meaningful differences. Too small, and you'll miss real improvements. Too large wastes time and traffic.
Key factors:
- Baseline metric value: Current task completion rate or response quality score
- Minimum detectable effect (MDE): The smallest improvement worth caring about (e.g., +2% accuracy)
- Statistical power: Typically 80% (chance of detecting a real effect)
- Significance level: Usually 5% (tolerance for false positives)
Analogy: Like a scale measuring weight. Too imprecise (small sample), you can't tell if something got lighter. Too precise but slow (huge sample), you waste time.
Key Takeaway: Proper randomization eliminates bias, while appropriate sample size ensures you have enough statistical power to detect real differences in your AI product's performance without wasting resources on unnecessarily large experiments.