Course contentsShow
← Data Science
Lesson 1581 of 2,14538. Bayesian Data AnalysisFree lesson

Setting Priors for A/B Tests

Choose appropriate prior distributions for conversion rates and other metrics in A/B testing contexts.

Setting Priors for A/B Tests

You'll learn: How to choose appropriate prior distributions for conversion rates and other metrics when conducting Bayesian A/B tests.

The Core Challenge

When running a Bayesian A/B test, you must specify a prior distribution before seeing any data. This prior represents your beliefs about the metric (like conversion rate) before the experiment runs. The question is: what shape and parameters should that prior have?

Choosing Priors for Conversion Rates

For conversion rates (proportions), the Beta distribution is the natural choice because it:

  • Lives between 0 and 1 (like all proportions)
  • Is conjugate with the binomial likelihood (making math tractable)
  • Flexibly represents different beliefs via two parameters (α, β)

Common Prior Strategies

Uninformative Prior: Use Beta(1, 1) (uniform) when you have zero prior knowledge. Every conversion rate is equally likely.

Weakly Informative Prior: Use Beta(2, 20) or similar if you expect roughly 10% conversion but aren't certain. This gently guides inference without overpowering the data.

Informative Prior: Use historical data. If last quarter showed 1,200 conversions from 10,000 visitors, try Beta(1200, 8800). This strongly influences early results but gets updated as data accumulates.

The Practical Balance

Think of priors like a "head start" in a race. Weak priors give your hypothesis a tiny nudge; strong priors require more data to overcome. For A/B tests, slightly informative priors (like Beta(5, 95) for a ~5% baseline) are often ideal—they stabilize early estimates without biasing final conclusions.

Key Takeaway: Choose priors that honestly reflect your pre-experiment beliefs: use Beta(1,1) for total ignorance, weakly informative Beta(α,β) matching historical rates for typical cases, and strong informative priors only when you have solid previous data to justify them.