AdaBoost Algorithm
What you'll learn: How AdaBoost adaptively re-weights training examples to force subsequent weak learners to focus on previously misclassified instances.
The Core Idea
AdaBoost (Adaptive Boosting) builds an ensemble by training weak learners sequentially, where each new learner pays special attention to the examples the previous learners got wrong. Instead of treating all training instances equally, AdaBoost maintains a weight for each example, increasing weights for misclassified points and decreasing weights for correctly classified ones.
Think of it like a teacher adjusting their lesson plan: if most students understand Topic A but struggle with Topic B, the teacher spends more time on Topic B in the next session.
How Re-weighting Works
- Initialize weights: Every training example starts with equal weight (typically
1/nwherenis dataset size) - Train a weak learner: Build a simple classifier on the weighted dataset
- Evaluate performance: Calculate how many examples were misclassified
- Update weights:
- Increase weights for misclassified examples (making them "heavier")
- Decrease weights for correctly classified examples
- Repeat: The next weak learner sees a modified version of the data where hard examples have more influence
The beauty is that each learner automatically focuses computational effort where previous learners struggled, creating a complementary team of specialists.
The Adaptive Part
The term "adaptive" refers to this dynamic re-weighting. The algorithm adapts to its own mistakes iteration by iteration, ensuring that no group of examples gets consistently ignored. By the end, you have multiple weak learners that collectively cover the entire problem space.
Key Takeaway: AdaBoost adaptively re-weights training examples after each iteration, amplifying the importance of misclassified instances so subsequent weak learners focus on the "hard cases" the ensemble hasn't yet mastered.