Validating Surrogate Metrics
What you'll learn: How to verify that your surrogate metric actually predicts the business metric you care about using historical data analysis.
Why Validation Matters
You've chosen a surrogate metric (like click-through rate) to stand in for a business metric (like revenue) because it's faster to measure. But here's the critical question: does your surrogate actually work?
Think of it like using body temperature to diagnose illness. Before doctors rely on fever as a surrogate for infection, they need historical evidence showing that elevated temperature reliably predicts actual bacterial or viral presence. The same principle applies to your metrics.
Core Validation Techniques
Historical Correlation Analysis
Examine past data where both your surrogate and business metrics were measured. Calculate the correlation between them across different time periods, user segments, and product changes. A strong, consistent correlation (typically r > 0.7) suggests your surrogate might be reliable.
Segment-Level Agreement
Check if the surrogate metric moves in the same direction as the business metric across different user segments. If clicking more predicts revenue for new users but not for power users, your surrogate may be unreliable.
Lag Analysis
Determine the time delay between surrogate metric changes and business metric changes. If clicks increase today, does revenue increase tomorrow? Next week? Understanding this lag helps you interpret A/B test results correctly.
Sensitivity Testing
Review historical experiments or product changes. When the business metric moved significantly, did the surrogate metric also move? False negatives (surrogate didn't budge when business metric did) reveal validation problems.
Key Takeaway: Never assume a surrogate metric is valid—use historical data to verify strong correlation, consistent directional agreement across segments, predictable time lags, and sensitivity to real business changes before relying on it in A/B tests.