Course contentsShow
← Machine Learning and Deep Learning
Lesson 3073 of 3,53867. A/B Testing and Deployment StrategiesFree lesson

Choosing Evaluation Metrics for A/B Tests

Select appropriate primary and secondary metrics that align with business objectives and user experience.

Choosing Evaluation Metrics for A/B Tests

What you'll learn: How to select primary and secondary metrics for A/B tests that meaningfully reflect both business goals and user experience.

The Challenge of Metric Selection

Choosing the right metrics for your A/B test is like choosing the right compass for a journey—pick poorly, and you'll optimize for the wrong destination. While your model might have excellent offline accuracy, the metrics that matter in production A/B tests must directly connect to business outcomes and user satisfaction.

Primary vs Secondary Metrics

Your primary metric is the single north star you're optimizing for—the metric that will determine whether variant B "wins." This should align tightly with your core business objective. For a recommendation system, this might be click-through rate, revenue per user, or engagement time.

Secondary metrics serve as guardrails and provide context. They help you understand whether improvements in your primary metric come at the cost of other important dimensions. For example, if your new model increases clicks (primary) but also increases page load time or decreases user retention (secondary), you've discovered a critical trade-off.

Alignment with Business Objectives

Your metrics must translate model performance into business impact. Precision@5 might matter in the lab, but in production, ask: does this A/B test increase revenue? Improve retention? Reduce churn? Connect every metric to a stakeholder's question.

Similarly, user experience metrics—like task completion rate, time-to-result, or perceived relevance—ensure you're not sacrificing usability for marginal statistical gains.

Practical Selection Strategy

Choose 1 primary metric (your decision-maker), 2-4 secondary metrics (guardrails and context), and ensure all are measurable within your test duration. Avoid metrics with delayed ground truth unless you have reliable proxy metrics.

Key Takeaway: Select one primary metric that directly reflects your business goal, supported by secondary metrics that protect user experience and provide contextual insights—this creates a balanced evaluation framework that prevents optimizing for hollow victories.