AdaBoost Weight Updates and Sample Reweighting
You'll learn the mathematical formulas behind how AdaBoost adjusts sample weights and combines weak learner predictions into a strong ensemble.
The Core Mechanism
AdaBoost works by training weak learners sequentially, each focusing more on previously misclassified samples. But how exactly does it decide which samples need more attention, and how much weight should each weak learner get in the final vote?
The magic lies in two formulas that work together like a feedback loop.
The Weight Update Formula
After each weak learner makes predictions, AdaBoost:
-
Calculates the weighted error rate: The sum of weights for misclassified samples divided by total weight
-
Computes the learner's authority (alpha):
α = 0.5 × ln((1 - error) / error)- Low error → high alpha (trusted learner)
- Error near 0.5 (random guessing) → alpha near 0 (ignored)
-
Updates each sample's weight:
- Misclassified:
new_weight = old_weight × exp(α) - Correctly classified:
new_weight = old_weight × exp(-α)
- Misclassified:
Think of it like a teacher giving harder problems to students who got questions wrong, while letting successful students coast a bit.
Combining Predictions
The final prediction isn't a simple majority vote—it's a weighted vote where each weak learner's vote is multiplied by its alpha value. Better learners get louder voices.
The weights are then normalized (divided by their sum) so they form a proper probability distribution for the next round.
Key Takeaway: AdaBoost's power comes from exponentially increasing weights on misclassified samples and weighting learner votes by their accuracy, creating a strong classifier from multiple weak ones through mathematical precision.