Survivorship Bias and Attrition
You'll learn how differential dropout between treatment and control groups can invisibly distort your A/B test results by changing who remains in each group over time.
The Core Problem
Imagine you're testing a new feature that makes your app more complex but powerful. You run a two-week A/B test and find that users in the treatment group love it—engagement is way up! But here's the catch: casual users found the new feature confusing and quit early, leaving only power users in your treatment group. Meanwhile, your control group still has a mix of casual and power users.
You're no longer comparing apples to apples. The groups started identical (thanks to randomization), but differential attrition—when participants drop out at different rates or for different reasons between groups—has changed the composition of who you're measuring.
Why This Matters in A/B Testing
Survivorship bias occurs when you only analyze the survivors—those who stuck around. If your treatment causes certain types of users to leave (or stay) more than the control does, your final metrics reflect:
- Who remained, not the original random sample
- A biased comparison between fundamentally different populations
This undermines the entire point of randomization. Even if you started with perfectly balanced groups, attrition can introduce selection bias after the fact.
Real Example
Testing a paywall feature: engaged users might subscribe (stay), while casual users leave. Your treatment group becomes disproportionately high-value users. When you measure "average revenue per user," it looks amazing—but only because you're measuring survivors, not the impact on your original user base.
Key Takeaway: Always track and compare dropout rates between groups; if attrition differs significantly or systematically, your treatment effect estimates may be biased because you're no longer measuring the same populations you randomized.