Handling Network Effects and Interference
You'll learn: How to recognize and address scenarios where treating one user in an A/B test affects others, breaking fundamental experimental assumptions.
The SUTVA Violation Problem
In classic A/B testing, we assume SUTVA (Stable Unit Treatment Value Assumption): the treatment applied to one user shouldn't affect another user's outcome. But in real-world ML systems—especially social platforms, marketplaces, or recommendation engines—this often breaks down.
Network effects occur when your treatment "spills over." For example:
- A ride-sharing app tests a driver incentive (treatment). More drivers accept rides, so control-group passengers get faster pickups too.
- A social network shows promoted posts to treatment users. Their friends in the control group see these posts shared organically.
- A search ranking model change affects what results users click, which changes what future users see (through popularity signals).
Why This Matters
When interference exists, your A/B test results become biased. You might underestimate the true effect (if treatment helps control users) or overestimate it (if treatment harms control users through resource competition). Your variance estimates are also wrong because observations aren't truly independent.
Mitigation Strategies
- Cluster randomization: Assign entire groups (cities, communities, time periods) to treatment/control rather than individuals
- Ego-network splitting: Isolate social graphs so treatment and control users don't interact
- Switchback experiments: Alternate treatment over time for shared-resource systems
- Modeling the interference: Use techniques like "two-sided tests" that explicitly measure spillover effects
- Accept limitations: Report results with caveats about potential interference when isolation isn't feasible
Key Takeaway: When users interact or compete for shared resources, standard A/B testing assumptions break down—use cluster randomization, isolation strategies, or specialized experimental designs to get valid causal estimates.