Course contentsShow
← Data Science
Lesson 1521 of 2,14537. A/B TestingFree lesson

Risks of Optimizing for Surrogates

Understand Goodhart's Law and how over-optimizing surrogates can harm true business goals with real-world examples.

Risks of Optimizing for Surrogates

You'll learn how over-optimizing surrogate metrics can backfire and damage the true business goals you're trying to improve.

The Core Problem: Goodhart's Law

When you optimize heavily for a surrogate metric, you risk falling victim to Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

Remember, surrogate metrics are proxies for your true business goals—they're easier to measure but not what you actually care about. The danger comes when teams optimize so aggressively for the surrogate that they find ways to "game" it without improving (or even while harming) the real objective.

Real-World Examples

Example 1: Click-Through Rate (CTR)
A team optimizes for CTR as a surrogate for user engagement. They discover that sensationalized, clickbait headlines dramatically boost clicks. CTR soars—but users quickly bounce, satisfaction drops, and long-term revenue falls. The surrogate improved while the business metric deteriorated.

Example 2: Video Autoplay
A streaming platform uses "videos started" as a surrogate for engagement. They implement aggressive autoplay features that inflate this metric artificially. Users feel annoyed by the loss of control, leading to increased churn—the opposite of the intended goal.

Example 3: Email Open Rates
Marketing optimizes for email open rates by using misleading subject lines. Opens increase, but unsubscribe rates and brand trust plummet.

Why This Happens

When you A/B test and iterate exclusively on a surrogate, you find every possible way to move that needle—including ways that exploit the gap between the surrogate and what truly matters. The surrogate-business metric relationship breaks down under pressure.

Key Takeaway: Always validate that improvements in surrogate metrics translate to gains in your true business objectives. Monitor both simultaneously, and be wary when a surrogate moves dramatically without corresponding business metric movement—you may be optimizing the wrong thing.