Updating Beliefs with Test Data
What you'll learn: How to apply Bayes' theorem to continuously update your posterior distributions as new A/B test data arrives, allowing decisions before the test formally "ends."
The Core Idea
In Bayesian A/B testing, you don't wait for a fixed sample size before looking at results. Instead, you start with prior beliefs (your prior distribution), observe some data, calculate a posterior, and then that posterior becomes your new prior when the next batch of data arrives. Each update gives you a refined probability distribution over conversion rates or other metrics.
Think of it like updating your weather app: your morning forecast (prior) gets refined by afternoon temperature readings (likelihood), producing an updated forecast (posterior). When evening data arrives, that afternoon forecast becomes your new starting point.
How Sequential Updating Works
- Start: Begin with your prior distribution for variants A and B (e.g., Beta distributions)
- Observe: Collect data for a period (say, one day's worth of conversions)
- Update: Apply Bayes' theorem to compute the posterior using that data
- Repeat: Tomorrow's prior = today's posterior; add new data and update again
For a Beta-Binomial model tracking conversions:
- Prior:
Beta(α₀, β₀) - After observing s successes and f failures
- New posterior:
Beta(α₀ + s, β₀ + f)
This accumulation property makes Bayesian updating beautifully simple for conjugate priors.
Why This Matters for A/B Tests
Unlike frequentist tests where peeking inflates error rates, Bayesian inference naturally handles sequential looks. Your posterior credible intervals validly represent uncertainty at any moment, letting you monitor "probability B beats A" continuously and stop when confident.
Key Takeaway: Bayesian A/B testing updates beliefs incrementally as data arrives—yesterday's posterior becomes today's prior, enabling valid continuous monitoring without the multiple testing penalties of frequentist approaches.