Pre-processing: Balancing Training Data
What you'll learn: How to balance demographic representation in training data to reduce bias before model training begins.
When your training data over-represents one group and under-represents others, your model learns skewed patterns that perpetuate unfairness. Pre-processing techniques fix this before training starts—like balancing ingredients before baking rather than trying to fix a lopsided cake afterward.
Core Balancing Techniques
Resampling adjusts the quantity of examples per group:
- Oversampling: duplicate or add more examples from under-represented groups
- Undersampling: reduce examples from over-represented groups
- Stratified sampling: ensure proportional representation across groups
Reweighting keeps all data but assigns importance scores. Instead of removing examples, you give under-represented groups higher weights during training. Think of it as turning up the volume on quieter voices rather than removing loud ones.
Synthetic data generation creates new, realistic examples for under-represented groups using techniques you learned earlier. This expands your dataset without just copying existing examples, reducing overfitting risk.
Practical Considerations
- Identify protected attributes first (gender, race, age, etc.) using your data classification skills
- Measure initial imbalance with demographic parity metrics
- Choose the right technique: oversample when you have little data, undersample when you have plenty, reweight when you want to keep everything
- Validate fairness metrics after balancing to confirm improvement
- Document all transformations in your audit trail
Balancing isn't a silver bullet—biased labels or stereotypical associations may still exist—but it's a crucial first step in the fairness pipeline.
Key Takeaway: Pre-processing balances demographic representation through resampling, reweighting, or synthetic generation, addressing unfairness at the data level before models ever train.