Root Cause Analysis for Drift
What You'll Learn
You'll learn systematic techniques to diagnose whether model performance degradation stems from data drift, concept drift, or data quality issues—and how to tell them apart.
The Detective Work of Production ML
When your model's performance drops in production, you're facing a mystery: what changed? Root cause analysis (RCA) for drift is your investigation toolkit to pinpoint the culprit among three usual suspects.
The Three Suspects
Data Drift (Covariate Shift): Your input features have changed distribution, but the relationship between features and target remains stable. Think: a retail model trained on summer data now sees winter shopping patterns.
Concept Drift: The underlying relationship between features and target has changed. Same inputs now mean different things—like "good credit score" predicting loan default differently during an economic crisis.
Data Quality Issues: Corrupted pipelines, broken sensors, or preprocessing bugs. Features arrive malformed, missing, or incorrectly transformed.
Diagnostic Strategy
-
Check data quality first: Validate schema, null rates, range violations, and encoding errors. If your feature pipeline broke, drift metrics are meaningless.
-
Measure input drift: Use statistical tests (KS, PSI) on features against your reference distribution. High drift in multiple features suggests covariate shift.
-
Assess performance with drift severity: If heavily drifted features don't correlate with performance drops, suspect concept drift instead.
-
Analyze prediction-target relationships: Plot model scores against actual outcomes. If calibration breaks (same score, different conversion rate), concept drift is likely.
-
Segment analysis: Break down drift and performance by feature subgroups. Isolated issues point to data quality; widespread shifts suggest genuine drift.
Key Takeaway: Root cause analysis requires triangulating data quality checks, input drift metrics, and performance degradation patterns—no single metric tells the whole story, so investigate systematically from data integrity through input distributions to prediction-outcome relationships.