Bayesian vs Frequentist A/B Testing
You'll learn the fundamental philosophical and practical differences between Bayesian and frequentist approaches when running A/B tests.
The Core Philosophical Divide
When you run an A/B test, you need to decide: Is the conversion rate a fixed unknown value, or is it a distribution of possible values?
Frequentist A/B testing treats the true conversion rate as a fixed (but unknown) parameter. You collect data, compute a p-value, and either reject or fail to reject the null hypothesis. The p-value tells you: "If there were truly no difference, how surprising would my data be?" You never make probability statements about the conversion rate itself.
Bayesian A/B testing treats the conversion rate as a random variable with a probability distribution. You start with a prior belief, update it with data to get a posterior distribution, and then make direct probability statements like "There's a 92% chance variant B is better than A."
Key Practical Differences
| Aspect | Frequentist | Bayesian |
|---|---|---|
| Output | p-value, confidence interval | Posterior distribution, credible interval |
| Interpretation | "If null true, data this extreme X% of time" | "Variant B better with 92% probability" |
| Stopping | Fixed sample size or sequential correction needed | Natural sequential updating, stop anytime |
| Prior knowledge | Ignored | Incorporated explicitly |
| Decision-making | Binary (significant/not) | Probabilistic (quantify uncertainty) |
Think of it this way: frequentists are like referees who only call fouls based on strict rules, while Bayesians are like advisors who give you odds and let you decide based on your risk tolerance.
Key Takeaway: Frequentist A/B testing emphasizes controlling long-run error rates with fixed procedures, while Bayesian A/B testing provides direct probability statements about which variant is better and naturally handles sequential testing through belief updating.