Course contentsShow
← Data Science
Lesson 1478 of 2,14537. A/B TestingFree lesson

Defining Success Metrics

Choosing primary metrics that align with business goals and defining secondary metrics for guardrails.

Defining Success Metrics

What you'll learn: How to choose the right metrics to measure whether your A/B test actually succeeded—and what guardrails prevent unintended harm.

Why Metrics Matter

After designing an A/B test with solid principles, you need to decide what you're measuring. Not all metrics are created equal. Your success metrics are the quantitative measures that tell you whether your experiment improved the business outcome you care about. Without clear metrics defined upfront, you're flying blind.

Primary Metrics: The North Star

Your primary metric (or success metric) must directly align with your business goal. If you're testing a new checkout flow, your primary metric might be "conversion rate" or "revenue per user." This is what you'll use to declare the test a winner or loser.

Think of the primary metric as your destination on a map—it's where you're trying to go. Choose one primary metric to avoid confusion and multiple-testing problems. It should be:

  • Measurable during the test period
  • Sensitive enough to detect realistic changes
  • Aligned with long-term business value

Secondary Metrics: Guardrails and Context

Secondary metrics protect you from winning the battle but losing the war. For example, you might increase purchases (primary metric) but also increase returns or customer complaints (secondary metrics).

Secondary metrics act as guardrails—they ensure your "win" doesn't cause harm elsewhere. They provide context: Did users spend more time confused? Did page load time increase?

Key Takeaway: Define one primary metric that directly measures your business goal, and choose secondary metrics as guardrails to catch unintended negative consequences—before you start the test.