Detecting Drift in Categorical Features
What you'll learn: How to monitor discrete variables for drift, accounting for new categories and frequency shifts that don't apply to continuous features.
Why Categorical Features Need Special Treatment
Continuous features use statistical distance metrics (like KL-divergence or KS tests), but categorical data requires different approaches. With discrete values, you're watching for two distinct problems: entirely new categories appearing (structural drift) and existing categories changing frequency (distributional drift).
Think of monitoring a "payment_method" feature in production. If users suddenly start using "crypto" when your training data only had "card," "cash," and "PayPal," that's a structural change your model never learned to handle. Even if no new categories appear, a shift from 80% card usage to 30% signals behavioral drift your model may struggle with.
Key Detection Strategies
1. Population Stability Index (PSI)
Measures how much the distribution of categories has shifted between reference (training) and current data:
PSI = Σ (current% - reference%) × ln(current% / reference%)
PSI > 0.1 suggests investigation needed; > 0.25 indicates significant drift.
2. Chi-Squared Test
Tests whether category frequencies differ significantly from expected distribution. More sensitive to rare categories than PSI.
3. New Category Detection
Simply track categories unseen during training—these automatically indicate drift regardless of statistical tests.
4. Frequency Ratio Monitoring
Track the ratio of top-k most common categories. If the top 3 categories dropped from 90% to 60% of samples, distribution shape changed dramatically.
Key Takeaway: Categorical drift detection requires tracking both what categories exist (structural changes) and how often they occur (frequency shifts), using specialized metrics like PSI and chi-squared tests rather than continuous-variable distance measures.