Unit of Randomization
You'll learn: How to choose the right unit (user, session, event, etc.) to randomize in your A/B test and why it matters for valid results.
What Is a Unit of Randomization?
The unit of randomization is the entity you assign to treatment or control groups in your A/B test. Think of it as deciding who or what gets flipped into variant A or variant B. Common choices include:
- User/Customer: Each individual person gets one experience
- Session: Each visit to your site gets randomized independently
- Device: Each phone/computer gets assigned
- Cluster: Groups like households, stores, or geographic regions
Why Does This Matter?
Your choice directly affects:
-
Statistical independence: Sessions from the same user aren't independent—they're correlated. If you randomize by session but analyze by user, your variance estimates will be wrong.
-
User experience consistency: Randomizing by session means the same user might see different versions on different visits, creating confusion.
-
Network effects: In social features, randomizing by user may "leak" treatment effects to control users who interact with treated users.
The Key Trade-Off
Randomizing by user gives cleaner results and consistent experience, but requires more users to detect effects. Randomizing by session gives you more experimental units (higher power), but risks violating independence assumptions and creates inconsistent experiences.
Rule of thumb: Use user-level randomization when possible for most web/app tests. Choose coarser units (clusters) when spillover effects exist, or finer units (sessions/pageviews) only when independence truly holds and consistency doesn't matter.
Key Takeaway: The unit of randomization determines both the validity of your statistical inference and the quality of user experience—choose the level that matches your hypothesis while respecting dependency structures in your data.