Class Weights and Cost-Sensitive Learning
What you'll learn: How to make your model care more about rare classes by adjusting the penalty for getting them wrong, without touching your dataset.
The Core Idea
So far you've seen resampling techniques that physically change your training data. Class weights take a different approach: they tell your model's loss function to punish mistakes on minority class examples more severely. Instead of creating more minority samples or removing majority ones, you're saying "every time you get a minority example wrong, it counts as if you got it wrong 10 times."
Think of it like grading a test where some questions are worth more points. A mistake on a rare disease diagnosis might cost 50 points, while missing a common cold costs only 1 point. The student (your model) learns to focus more attention on those high-stakes questions.
How It Works
Most algorithms use a loss function to measure prediction errors. Class weights multiply the loss for each class:
- Majority class (90% of data): weight = 0.5 → mistakes cost less
- Minority class (10% of data): weight = 5.0 → mistakes cost much more
During training, the optimizer sees these weighted errors and adjusts the model to reduce the more expensive mistakes first. This naturally shifts the decision boundary to better capture minority examples.
In Practice
Scikit-learn makes this easy with the class_weight parameter (available in logistic regression, SVMs, random forests, etc.). Setting class_weight='balanced' automatically computes weights inversely proportional to class frequencies.
Key Takeaway: Class weights let you apply cost-sensitive learning by modifying the loss function to penalize minority class errors more heavily, forcing the model to pay more attention to rare cases without altering the training data distribution.