Course contentsShow
← AI Engineering
Lesson 907 of 1,88622. Testing and CI/CD for AI SystemsFree lesson

Regression Detection in CI

Implement checks that fail builds when evaluation metrics drop below thresholds.

Regression Detection in CI

You'll learn how to automatically fail CI builds when your AI system's evaluation metrics fall below acceptable performance thresholds.

What is Regression Detection in CI?

Regression detection in CI means setting up automated quality gates that prevent degraded model performance from reaching production. Just like how traditional software CI fails when unit tests break, your AI CI pipeline should fail when accuracy, latency, or other key metrics drop below defined thresholds.

Think of it as a safety net: if your latest prompt change causes answer quality to drop from 85% to 60%, CI catches it before merge, not after users complain.

How It Works

After running your automated tests (unit, integration, E2E, and regression suite), your CI pipeline compares current metrics against baselines:

  1. Run evaluation suite — Execute your tests and collect metrics (accuracy, F1, latency, cost)
  2. Compare against thresholds — Check if metrics meet minimum requirements
  3. Fail the build — If any metric falls below threshold, mark CI run as failed
  4. Block deployment — Prevent merge or deployment until fixed

Example threshold checks in your CI pipeline:

# Pseudocode for CI check
if accuracy < 0.80:
    fail_build("Accuracy dropped below 80%")
if p95_latency > 3.0:
    fail_build("P95 latency exceeded 3s")
if cost_per_request > 0.05:
    fail_build("Cost per request too high")

Setting Thresholds

Base thresholds on your production baseline metrics, not aspirational goals. Allow small acceptable variations (e.g., 2-3% drops) but catch significant regressions. Store thresholds in configuration files alongside your tests.

Key Takeaway: Regression detection in CI automatically blocks deployments when evaluation metrics drop below acceptable thresholds, catching performance degradation before it reaches users.