Course contentsShow
← Machine Learning and Deep Learning
Lesson 3047 of 3,53866. Production Monitoring and Data DriftFree lesson

Root Cause Analysis for Drift

Techniques to determine whether performance degradation is due to data drift, concept drift, or data quality issues.

Root Cause Analysis for Drift

What You'll Learn

You'll learn systematic techniques to diagnose whether model performance degradation stems from data drift, concept drift, or data quality issues—and how to tell them apart.

The Detective Work of Production ML

When your model's performance drops in production, you're facing a mystery: what changed? Root cause analysis (RCA) for drift is your investigation toolkit to pinpoint the culprit among three usual suspects.

The Three Suspects

Data Drift (Covariate Shift): Your input features have changed distribution, but the relationship between features and target remains stable. Think: a retail model trained on summer data now sees winter shopping patterns.

Concept Drift: The underlying relationship between features and target has changed. Same inputs now mean different things—like "good credit score" predicting loan default differently during an economic crisis.

Data Quality Issues: Corrupted pipelines, broken sensors, or preprocessing bugs. Features arrive malformed, missing, or incorrectly transformed.

Diagnostic Strategy

  1. Check data quality first: Validate schema, null rates, range violations, and encoding errors. If your feature pipeline broke, drift metrics are meaningless.

  2. Measure input drift: Use statistical tests (KS, PSI) on features against your reference distribution. High drift in multiple features suggests covariate shift.

  3. Assess performance with drift severity: If heavily drifted features don't correlate with performance drops, suspect concept drift instead.

  4. Analyze prediction-target relationships: Plot model scores against actual outcomes. If calibration breaks (same score, different conversion rate), concept drift is likely.

  5. Segment analysis: Break down drift and performance by feature subgroups. Isolated issues point to data quality; widespread shifts suggest genuine drift.

Key Takeaway: Root cause analysis requires triangulating data quality checks, input drift metrics, and performance degradation patterns—no single metric tells the whole story, so investigate systematically from data integrity through input distributions to prediction-outcome relationships.