Selection Bias in Opt-In Tests
You'll learn: Why letting users choose their test variant breaks randomization and ruins your ability to draw valid causal conclusions.
The Core Problem
In a proper A/B test, users are randomly assigned to control or treatment. This ensures the groups are comparable on average—any observed difference in outcomes can be attributed to the treatment itself.
But imagine you run an "opt-in" test where users can choose whether to use the new feature. Now randomization is gone. The people who opt in are fundamentally different from those who don't—they're likely more engaged, more tech-savvy, or simply more curious. This self-selection creates selection bias.
Why It Breaks Causal Inference
When users choose their variant, you're no longer comparing apples to apples. If the opt-in group shows better metrics, is it because:
- The feature actually works? OR
- The people who opted in were already your best users?
You can't tell. The treatment and the type of user are now confounded. Any causal claim is invalid.
Real-World Example
Suppose you launch a premium analytics dashboard and let users opt in. Power users flock to it, and you see a 20% increase in retention. Success? Not necessarily. Those power users might have stayed anyway. You've measured correlation, not causation.
The DAG Perspective
In DAG terms, user characteristics (engagement, expertise) now influence both treatment assignment and outcome—they're an unmeasured confounder. This opens a backdoor path, violating the randomization that would otherwise block it.
Key Takeaway: Allowing users to choose variants destroys randomization, introduces selection bias, and makes it impossible to isolate the true causal effect of your treatment. Always assign randomly when measuring impact.