Preventing Target Leakage in Target Encoding
What You'll Learn
You'll learn how to safely use target encoding without accidentally giving your model unfair access to future information it shouldn't have during training.
The Problem: Peeking at the Answer
Remember target encoding from the previous lesson—replacing categories with their average target values? There's a dangerous trap: if you calculate these averages using the same data you're training on, your model sees information it wouldn't have in real predictions. This is called target leakage.
Think of it like studying for a test: if you memorize answers while looking at the answer key, you'll do great on that exact test but fail when questions change slightly. Your model will overfit.
The Solution: Cross-Validation Folds
The safest approach uses cross-validation folds:
- Split your training data into K folds (say, 5 groups)
- For each fold, calculate target encodings using only the other folds
- Apply those encodings to the current fold
This way, each row gets encoded using statistics it wasn't part of calculating—mimicking real-world conditions where you encode new data using historical patterns.
Adding Smoothing
For rare categories with few examples, target statistics are unreliable. Smoothing blends the category average with the global average:
encoded_value = (category_mean × n + global_mean × weight) / (n + weight)
Where n is the count of that category. This prevents extreme encodings from small sample sizes.
Why It Matters
Without these precautions, your model will show artificially high training accuracy but fail miserably on new data. Cross-validation encoding ensures your encodings reflect only past information, and smoothing adds statistical stability.
Key Takeaway: Prevent target leakage in target encoding by using cross-validation folds to compute encodings (never on the same data being encoded) and apply smoothing to stabilize rare categories.