Course contentsShow
← Data Science
Lesson 1745 of 2,14541. Marketing and Growth AnalyticsFree lesson

Holdout Groups and Test Design

Learn to design incrementality tests using control groups that don't receive the marketing treatment.

Holdout Groups and Test Design

What you'll learn: How to structure an incrementality test using control groups that never see your marketing campaign, so you can measure its true causal impact.

The Core Concept

A holdout group is a randomly selected set of users who are excluded from receiving a marketing treatment (like an ad, email, or promotion). By comparing their behavior to a similar treatment group that does receive it, you can measure the incremental lift your campaign actually caused — not just correlation.

Think of it like a medical trial: you can't know if the medicine works unless some people take a placebo. In marketing, your "placebo" is the absence of your campaign.

Test Design Essentials

Random Assignment

Split users before the campaign starts. Randomization ensures the two groups are comparable in all ways except exposure to your treatment.

Sizing Your Holdout

Typically 5-20% of your audience. Too small = noisy results. Too large = you're leaving money on the table by not marketing to them.

What to Measure

Compare conversion rates, revenue, engagement, or any business metric between groups. The difference is your incremental impact.

Duration

Run the test long enough to capture your full conversion window and account for carryover effects (adstock from previous lessons). Typically 2-4 weeks minimum.

Why This Beats Attribution

Attribution tells you which touchpoint got credit. Holdout tests tell you whether the campaign mattered at all. You might find that 30% of your "attributed" conversions would have happened anyway — that's baseline, not incrementality.

Key Takeaway: Holdout groups are the gold standard for measuring true marketing incrementality; randomly withhold treatment from a control group and compare outcomes to isolate causal impact.