Detecting Concept Drift with Model Performance
What you'll learn: How to use performance metric degradation over time to catch when the feature-target relationship has fundamentally changed.
The Performance Signal
While statistical tests (like those from lesson 3029) can detect data drift, the most direct way to confirm concept drift is watching your model's performance metrics decline over time windows. When accuracy drops, F1-score decreases, or RMSE climbs steadily, the relationship your model learned between features and targets may have shifted—even if the input distribution looks similar.
Core Monitoring Approach
Track key metrics on recent predictions where you've obtained true labels:
- Classification: Monitor accuracy, precision, recall, F1-score
- Regression: Track RMSE, MAE, R²
Compare these metrics across rolling time windows (e.g., daily, weekly). A significant, sustained drop signals concept drift. For example, if your fraud model's F1-score was 0.85 last month but drops to 0.72 this week, fraudsters may have changed tactics—same features, different patterns.
Why This Works
Unlike input drift detection (lesson 3027), which only looks at feature distributions, performance monitoring directly measures what matters: is the model still making good predictions? It catches concept drift even when features look statistically unchanged but their relationship to the target has evolved.
The Limitation
This approach requires ground truth labels with reasonable latency. You can't detect drift instantly—you need actual outcomes. That's why it complements (not replaces) proxy metrics and input drift detection for real-time alerting.
Key Takeaway: Performance degradation over sliding time windows (accuracy ↓, F1 ↓, RMSE ↑) is the gold standard for confirming concept drift, directly measuring whether the feature-target relationship your model relies on has changed.