Course contentsShow
← AI Engineering
Lesson 870 of 1,88621. Human Evaluation and FeedbackFree lesson

Choosing Metrics for AI A/B Tests

Define primary and secondary metrics that capture both model performance and user experience outcomes.

Choosing Metrics for AI A/B Tests

What you'll learn: How to select the right primary and secondary metrics that measure both your AI model's technical performance and real user experience in A/B tests.

Why Metrics Matter in AI A/B Testing

When you A/B test an AI feature, you're not just measuring if the model is "better"—you're measuring if it actually improves the user experience and business outcomes. Choosing the wrong metrics means you might optimize for technical excellence while users suffer, or vice versa.

Primary vs Secondary Metrics

Primary metrics are your north star—the single most important measure of success. For an AI feature, this should typically be a user experience or business metric, not a model metric. Examples:

  • Task completion rate (did users accomplish their goal?)
  • User satisfaction rating
  • Time to completion
  • Conversion rate or revenue impact

Secondary metrics provide supporting context and guardrails. These include:

  • Model performance metrics: accuracy, latency, token usage, error rates
  • User engagement signals: feature adoption, retry rates, feedback sentiment
  • Guardrail metrics: inappropriate content flags, user complaints, escalations

Think of it like driving: your primary metric is "Did we arrive safely and on time?" while secondary metrics are "fuel efficiency" and "engine temperature."

Balancing Model and User Outcomes

A faster model (technical win) that frustrates users with terse responses is a failure. A highly accurate model that takes 30 seconds to respond might lose users. Always anchor your primary metric in what users actually care about, then use secondary metrics to understand why the primary metric moved and to catch unintended consequences.

Key Takeaway: Define one primary metric focused on user experience or business value, supported by secondary metrics that capture model performance and potential negative side effects—never optimize for technical metrics alone.