Output Drift and Prediction Distribution Shifts
What you'll learn: How to detect degrading model performance by monitoring changes in what your model predicts, not just what it receives.
The Core Idea
While Input Drift (lesson 3027) focuses on changes in what comes in, Output Drift monitors changes in what comes out—your model's predictions. Even when input features look stable, your model's output distribution can shift, signaling that something is wrong with how the model is making decisions.
Think of it like a weather forecaster: if they suddenly start predicting rain 80% of the time when historically it was only 20%, something's off—even if the temperature and pressure readings (inputs) haven't changed much.
Why Output Drift Matters
Output drift is often an early warning signal before you have ground truth labels. In production, you might not know the true answer for hours, days, or weeks (recall from lesson 3017: the feedback loop challenge). But you immediately know what your model predicted and how confident it was.
What to Monitor
- Prediction class distribution: Are you suddenly predicting class A much more than before?
- Confidence scores: Is your model becoming less certain overall?
- Score distributions: Are predicted probabilities clustering differently?
For example, if your fraud detection model historically flagged 2% of transactions but suddenly flags 15%, that's output drift—investigate before relying on those predictions.
Connecting to Earlier Concepts
You can apply the same drift detection techniques from lessons 3029-3031 (statistical tests, distance metrics) to your output distributions instead of input features.
Key Takeaway: Output drift detection monitors your model's prediction distributions and confidence levels as immediate proxies for performance degradation, catching problems before ground truth labels arrive.