Course contentsShow
← Data Science
Lesson 1582 of 2,14538. Bayesian Data AnalysisFree lesson

Updating Beliefs with Test Data

Apply Bayes' theorem to update posterior distributions as A/B test data accumulates over time.

Updating Beliefs with Test Data

What you'll learn: How to apply Bayes' theorem to continuously update your posterior distributions as new A/B test data arrives, allowing decisions before the test formally "ends."

The Core Idea

In Bayesian A/B testing, you don't wait for a fixed sample size before looking at results. Instead, you start with prior beliefs (your prior distribution), observe some data, calculate a posterior, and then that posterior becomes your new prior when the next batch of data arrives. Each update gives you a refined probability distribution over conversion rates or other metrics.

Think of it like updating your weather app: your morning forecast (prior) gets refined by afternoon temperature readings (likelihood), producing an updated forecast (posterior). When evening data arrives, that afternoon forecast becomes your new starting point.

How Sequential Updating Works

  1. Start: Begin with your prior distribution for variants A and B (e.g., Beta distributions)
  2. Observe: Collect data for a period (say, one day's worth of conversions)
  3. Update: Apply Bayes' theorem to compute the posterior using that data
  4. Repeat: Tomorrow's prior = today's posterior; add new data and update again

For a Beta-Binomial model tracking conversions:

  • Prior: Beta(α₀, β₀)
  • After observing s successes and f failures
  • New posterior: Beta(α₀ + s, β₀ + f)

This accumulation property makes Bayesian updating beautifully simple for conjugate priors.

Why This Matters for A/B Tests

Unlike frequentist tests where peeking inflates error rates, Bayesian inference naturally handles sequential looks. Your posterior credible intervals validly represent uncertainty at any moment, letting you monitor "probability B beats A" continuously and stop when confident.

Key Takeaway: Bayesian A/B testing updates beliefs incrementally as data arrives—yesterday's posterior becomes today's prior, enabling valid continuous monitoring without the multiple testing penalties of frequentist approaches.