Interpreting A/B Test Results
What you'll learn: How to correctly interpret A/B test outcomes by distinguishing statistical significance from practical impact and avoiding common interpretation traps.
Beyond the P-Value
You've run your A/B test comparing model variants and got p < 0.05. Victory? Not so fast. Interpreting results requires understanding confidence intervals, practical significance, and recognizing misleading patterns.
Confidence Intervals: The Range of Truth
A confidence interval tells you the plausible range for the true effect size. A 95% CI of [0.5%, 2.3%] for CTR improvement means:
- The effect is statistically significant (doesn't include zero)
- But the actual improvement might be as small as 0.5%
Wide intervals signal uncertainty—you may need more data even if p < 0.05.
Statistical vs Practical Significance
This is the critical distinction:
- Statistical significance: The effect is unlikely due to chance (p-value)
- Practical significance: The effect matters for your business
With enough data, a 0.01% accuracy improvement can be "statistically significant" but costs $50K in compute monthly. Is it worth deploying? That's practical significance—a business decision, not a statistical one.
Common Pitfalls to Avoid
Peeking: Checking results repeatedly and stopping when significant inflates false positives. Use sequential testing methods instead (covered earlier).
Cherry-picking metrics: Testing 20 metrics and highlighting the one that's significant. Apply multiple testing corrections.
Ignoring directionality: A significant result in the wrong direction is still a failed experiment.
Confusing correlation with causation: Segment analysis ("model B wins for mobile users!") discovered after testing is exploratory, not confirmatory.
The Complete Picture
Always report: effect size, confidence interval, p-value, sample size, and test duration. Context matters—a 1% latency increase might doom a recommendation system but be negligible for batch processing.
Key Takeaway: Statistical significance proves an effect exists; practical significance and confidence intervals tell you whether it matters and how certain you should be—interpret both before making deployment decisions.