Course contentsShow
← Data Science
Lesson 1750 of 2,14541. Marketing and Growth AnalyticsFree lesson

Common Pitfalls and Limitations

Identify issues like spillover effects, selection bias, and insufficient power that compromise incrementality tests.

Common Pitfalls and Limitations

You'll learn: How spillover effects, selection bias, and insufficient power can undermine your incrementality tests and lead to incorrect conclusions about marketing impact.

The Threats to Valid Incrementality Measurement

Even with careful holdout groups and rigorous test design, incrementality tests face three major challenges that can compromise your results:

Spillover Effects

When your control group is contaminated by the treatment, you lose measurement integrity. Imagine running a geo-lift experiment where users in the "no ad" city see your ads on social media from friends in the "ad" city, or commute between test regions. Spillover dilutes the measured difference, making your campaigns appear less effective than they truly are.

Selection Bias

If your holdout groups aren't truly comparable to your treatment groups, your results are skewed from the start. Perhaps you assigned geo-regions based on convenience rather than randomization, and the holdout cities happen to be economically different. Or in ghost ad tests, the users shown PSAs had different browsing behaviors than those shown real ads. The measured "lift" reflects pre-existing differences, not true incrementality.

Insufficient Statistical Power

Running a test that's too small or too short produces unreliable results. You might declare a campaign incremental when it's just noise, or abandon an effective strategy because you couldn't detect its impact. Power depends on sample size, baseline conversion rates, expected effect size, and test duration—all must be properly calculated before you launch.

Real-World Implications

These pitfalls lead to misallocated budgets: you might scale channels that aren't actually incremental or cut effective ones. Always validate your test design assumptions, use randomization where possible, calculate required sample sizes upfront, and watch for external factors that could confound results.

Key Takeaway: Spillover effects, selection bias, and insufficient power are the primary threats to valid incrementality measurement—rigorous test design, randomization, and proper power calculations are essential to avoid false conclusions about marketing effectiveness.