Course contentsShow
← Computer Science
Lesson 2858 of 2,87263. Classical ML AlgorithmsFree lesson

AdaBoost Algorithm

Exploring how AdaBoost adaptively re-weights training examples to focus on misclassified instances in each iteration.

AdaBoost Algorithm

What you'll learn: How AdaBoost adaptively re-weights training examples to force subsequent weak learners to focus on previously misclassified instances.


The Core Idea

AdaBoost (Adaptive Boosting) builds an ensemble by training weak learners sequentially, where each new learner pays special attention to the examples the previous learners got wrong. Instead of treating all training instances equally, AdaBoost maintains a weight for each example, increasing weights for misclassified points and decreasing weights for correctly classified ones.

Think of it like a teacher adjusting their lesson plan: if most students understand Topic A but struggle with Topic B, the teacher spends more time on Topic B in the next session.

How Re-weighting Works

  1. Initialize weights: Every training example starts with equal weight (typically 1/n where n is dataset size)
  2. Train a weak learner: Build a simple classifier on the weighted dataset
  3. Evaluate performance: Calculate how many examples were misclassified
  4. Update weights:
    • Increase weights for misclassified examples (making them "heavier")
    • Decrease weights for correctly classified examples
  5. Repeat: The next weak learner sees a modified version of the data where hard examples have more influence

The beauty is that each learner automatically focuses computational effort where previous learners struggled, creating a complementary team of specialists.

The Adaptive Part

The term "adaptive" refers to this dynamic re-weighting. The algorithm adapts to its own mistakes iteration by iteration, ensuring that no group of examples gets consistently ignored. By the end, you have multiple weak learners that collectively cover the entire problem space.

Key Takeaway: AdaBoost adaptively re-weights training examples after each iteration, amplifying the importance of misclassified instances so subsequent weak learners focus on the "hard cases" the ensemble hasn't yet mastered.