Setting Priors for A/B Tests
You'll learn: How to choose appropriate prior distributions for conversion rates and other metrics when conducting Bayesian A/B tests.
The Core Challenge
When running a Bayesian A/B test, you must specify a prior distribution before seeing any data. This prior represents your beliefs about the metric (like conversion rate) before the experiment runs. The question is: what shape and parameters should that prior have?
Choosing Priors for Conversion Rates
For conversion rates (proportions), the Beta distribution is the natural choice because it:
- Lives between 0 and 1 (like all proportions)
- Is conjugate with the binomial likelihood (making math tractable)
- Flexibly represents different beliefs via two parameters (α, β)
Common Prior Strategies
Uninformative Prior: Use Beta(1, 1) (uniform) when you have zero prior knowledge. Every conversion rate is equally likely.
Weakly Informative Prior: Use Beta(2, 20) or similar if you expect roughly 10% conversion but aren't certain. This gently guides inference without overpowering the data.
Informative Prior: Use historical data. If last quarter showed 1,200 conversions from 10,000 visitors, try Beta(1200, 8800). This strongly influences early results but gets updated as data accumulates.
The Practical Balance
Think of priors like a "head start" in a race. Weak priors give your hypothesis a tiny nudge; strong priors require more data to overcome. For A/B tests, slightly informative priors (like Beta(5, 95) for a ~5% baseline) are often ideal—they stabilize early estimates without biasing final conclusions.
Key Takeaway: Choose priors that honestly reflect your pre-experiment beliefs: use Beta(1,1) for total ignorance, weakly informative Beta(α,β) matching historical rates for typical cases, and strong informative priors only when you have solid previous data to justify them.