Course contentsShow
← Data Science
Lesson 1583 of 2,14538. Bayesian Data AnalysisFree lesson

Probability of Being Best

Calculate and interpret the probability that each variant is the true winner of the A/B test.

Probability of Being Best

You'll learn how to calculate the probability that each variant in your A/B test is actually the best performing option.

The Core Idea

After running a Bayesian A/B test and updating your beliefs with test data, you don't just want to know the posterior distributions—you want to answer: "What's the chance variant A is better than variant B?" The Probability of Being Best directly answers this question by computing the probability that a given variant has the highest true conversion rate (or other metric) compared to all other variants.

How It Works

With your posterior distributions in hand, you simulate draws from each variant's posterior. For each simulation round:

  1. Draw one sample from variant A's posterior
  2. Draw one sample from variant B's posterior (and C, D, etc. if present)
  3. Check which variant had the highest value
  4. Repeat thousands of times

The proportion of times variant A "wins" is its probability of being best.

Example Interpretation

Suppose variant A has a 72% probability of being best, variant B has 25%, and variant C has 3%. This means:

  • Given your prior beliefs and observed data, there's a 72% chance A truly has the highest conversion rate
  • You can use this to make risk-informed decisions (e.g., "I'm comfortable launching A")

Unlike p-values, this probability directly quantifies what you care about: which variant is actually superior.

Key Takeaway: The probability of being best tells you the chance each variant is the true winner, giving you a clear, interpretable metric for decision-making in Bayesian A/B tests.