Probability of Being Best
You'll learn how to calculate the probability that each variant in your A/B test is actually the best performing option.
The Core Idea
After running a Bayesian A/B test and updating your beliefs with test data, you don't just want to know the posterior distributions—you want to answer: "What's the chance variant A is better than variant B?" The Probability of Being Best directly answers this question by computing the probability that a given variant has the highest true conversion rate (or other metric) compared to all other variants.
How It Works
With your posterior distributions in hand, you simulate draws from each variant's posterior. For each simulation round:
- Draw one sample from variant A's posterior
- Draw one sample from variant B's posterior (and C, D, etc. if present)
- Check which variant had the highest value
- Repeat thousands of times
The proportion of times variant A "wins" is its probability of being best.
Example Interpretation
Suppose variant A has a 72% probability of being best, variant B has 25%, and variant C has 3%. This means:
- Given your prior beliefs and observed data, there's a 72% chance A truly has the highest conversion rate
- You can use this to make risk-informed decisions (e.g., "I'm comfortable launching A")
Unlike p-values, this probability directly quantifies what you care about: which variant is actually superior.
Key Takeaway: The probability of being best tells you the chance each variant is the true winner, giving you a clear, interpretable metric for decision-making in Bayesian A/B tests.