Expected Loss and Decision Making
You'll learn how to use expected loss to quantify the financial or practical cost of choosing the wrong variant and decide when to stop your A/B test.
What is Expected Loss?
Knowing that variant B has a 70% probability of being best is useful, but it doesn't tell you how much worse variant A might be if you choose it instead. Expected loss answers: "If I pick this variant and it's wrong, how much will I lose on average?"
For each variant, you calculate the expected loss by looking at all possible outcomes (based on your posterior distribution) and computing the average difference between that variant and the best possible choice.
How It Informs Decisions
Imagine you're testing two checkout button colors. Variant A has 65% probability of being best, but if B is actually better, you'd only lose 500/day.
The expected losses might be:
- Expected loss of choosing A: 0.35 × 175/day
- Expected loss of choosing B: 0.65 × 32.50/day
Even though A has higher probability of being best, choosing B minimizes your expected loss—it's the safer, more economical decision.
Stopping Rules
You can set a threshold like "stop the test when expected loss < $10/day for the best variant." This gives you a concrete, business-aligned criterion instead of arbitrary probability thresholds.
Key Takeaway: Expected loss translates Bayesian probabilities into real costs, helping you make economically rational decisions about when to stop testing and which variant to deploy.