Prior Probability Shift
What you'll learn: How changes in class proportions (classification) or target distributions (regression) affect model performance in production, even when input features remain unchanged.
The Core Concept
Prior probability shift occurs when the distribution of your target variable changes between training and production—without any change to the input features themselves or how they relate to the target. In classification, this means the balance of classes shifts (e.g., fraud goes from 1% to 5% of transactions). In regression, the overall distribution of predicted values evolves (e.g., home prices shift upward across the board).
This is distinct from concept drift because the underlying relationships P(Y|X) stay the same—only the marginal distribution P(Y) changes. Think of it like a grocery store that trained a "busy vs not-busy" classifier during winter, when 30% of hours are busy. Come summer, 70% of hours are busy due to tourists, but the features that predict busyness (time of day, weather) work the same way.
Why It Matters
Your model's calibration and decision thresholds were optimized for the training distribution. When class proportions shift:
- Classification: Precision/recall trade-offs change; fixed thresholds become suboptimal
- Regression: Prediction ranges and error distributions shift; outlier detection may fail
Detection Strategy
Monitor your production label distribution (when available) or prediction distribution as a proxy. Compare against your reference training window using the drift detection techniques you've learned—KS tests for continuous targets, chi-square for categorical classes.
Key Takeaway: Prior probability shift means your target distribution has changed while feature-target relationships remain stable—requiring threshold recalibration or model retraining to maintain optimal decision-making.