Sample Ratio Mismatch (SRM)
You'll learn: How to detect when your A/B test groups have unexpectedly imbalanced sizes—a critical warning sign of implementation bugs or biased randomization.
What is Sample Ratio Mismatch?
Sample Ratio Mismatch occurs when the observed ratio of users in your control and treatment groups deviates significantly from the expected ratio based on your randomization design.
For example, if you designed a 50/50 split between control and treatment, you'd expect roughly equal group sizes. If you end up with 48,000 in control but 52,000 in treatment (a 48:52 ratio instead of 50:50), that's a potential SRM.
Why SRM Matters
SRM is a red flag that something went wrong during randomization or data collection:
- Implementation bugs: Maybe your randomization code has an off-by-one error or timestamp issues
- Biased assignment: Certain user types might be systematically excluded or included
- Data pipeline issues: Some users' data might not be logged correctly
- Pre-filtering problems: Different groups experiencing different dropout rates during assignment
Think of it like flipping a supposedly fair coin 100,000 times. If you get 60,000 heads and 40,000 tails, you'd suspect the coin isn't actually fair—or your counting method is broken.
Detecting SRM
You can use a chi-square goodness-of-fit test to check if observed group sizes match expected proportions. A significant p-value (typically < 0.01) suggests SRM and means you should pause the experiment and investigate before drawing conclusions.
Key Takeaway: Sample Ratio Mismatch is your experiment's smoke detector—when group sizes don't match your randomization design, stop and debug before trusting any results, because the underlying randomization may be broken.