Holdout Groups and Test Design
What you'll learn: How to structure an incrementality test using control groups that never see your marketing campaign, so you can measure its true causal impact.
The Core Concept
A holdout group is a randomly selected set of users who are excluded from receiving a marketing treatment (like an ad, email, or promotion). By comparing their behavior to a similar treatment group that does receive it, you can measure the incremental lift your campaign actually caused — not just correlation.
Think of it like a medical trial: you can't know if the medicine works unless some people take a placebo. In marketing, your "placebo" is the absence of your campaign.
Test Design Essentials
Random Assignment
Split users before the campaign starts. Randomization ensures the two groups are comparable in all ways except exposure to your treatment.
Sizing Your Holdout
Typically 5-20% of your audience. Too small = noisy results. Too large = you're leaving money on the table by not marketing to them.
What to Measure
Compare conversion rates, revenue, engagement, or any business metric between groups. The difference is your incremental impact.
Duration
Run the test long enough to capture your full conversion window and account for carryover effects (adstock from previous lessons). Typically 2-4 weeks minimum.
Why This Beats Attribution
Attribution tells you which touchpoint got credit. Holdout tests tell you whether the campaign mattered at all. You might find that 30% of your "attributed" conversions would have happened anyway — that's baseline, not incrementality.
Key Takeaway: Holdout groups are the gold standard for measuring true marketing incrementality; randomly withhold treatment from a control group and compare outcomes to isolate causal impact.