Ignoring Network Effects
You'll learn why independence assumptions break down when one user's experience influences another's—and how that can invalidate your A/B test results.
The Core Problem
In classic A/B testing, we assume that each user's outcome is independent: what happens to User A doesn't affect User B. But in many real-world platforms—social networks, marketplaces, communication apps—users interact with each other. When you randomize at the user level, someone in the treatment group might influence someone in the control group (or vice versa), creating treatment spillover.
Why This Matters
Imagine testing a new "share" button on a social platform. You show it to 50% of users (treatment) and hide it from the other 50% (control). But when treatment users share more content, control users see that content too. Their behavior changes—not because of their own assignment, but because of their friends' assignment. Your control group is contaminated.
This violates the independence assumption that underpins standard statistical tests. Your estimated treatment effect may be biased, and your confidence intervals are wrong.
Real-World Examples
- Rideshare apps: Drivers in treatment might reduce wait times for riders in control
- Marketplaces: More sellers (treatment) improve selection for all buyers
- Messaging apps: Features that change how one person messages affect the recipient
What You Can Do
- Cluster randomization: Randomize by groups (e.g., cities, social clusters) instead of individuals
- Ego-network analysis: Model and measure the spillover explicitly
- Accept limitations: Acknowledge when clean measurement isn't feasible
Key Takeaway: When users interact or share a common resource, individual randomization fails; treatment effects leak across groups, contaminating your control and biasing your results.