Course contentsShow
← Machine Learning and Deep Learning
Lesson 309 of 3,5386. Tree-Based MethodsFree lesson

AdaBoost Weight Updates and Sample Reweighting

Deriving the mathematical formulas for updating sample weights and combining weak learner predictions in AdaBoost.

AdaBoost Weight Updates and Sample Reweighting

You'll learn the mathematical formulas behind how AdaBoost adjusts sample weights and combines weak learner predictions into a strong ensemble.

The Core Mechanism

AdaBoost works by training weak learners sequentially, each focusing more on previously misclassified samples. But how exactly does it decide which samples need more attention, and how much weight should each weak learner get in the final vote?

The magic lies in two formulas that work together like a feedback loop.

The Weight Update Formula

After each weak learner makes predictions, AdaBoost:

  1. Calculates the weighted error rate: The sum of weights for misclassified samples divided by total weight

  2. Computes the learner's authority (alpha):

    • α = 0.5 × ln((1 - error) / error)
    • Low error → high alpha (trusted learner)
    • Error near 0.5 (random guessing) → alpha near 0 (ignored)
  3. Updates each sample's weight:

    • Misclassified: new_weight = old_weight × exp(α)
    • Correctly classified: new_weight = old_weight × exp(-α)

Think of it like a teacher giving harder problems to students who got questions wrong, while letting successful students coast a bit.

Combining Predictions

The final prediction isn't a simple majority vote—it's a weighted vote where each weak learner's vote is multiplied by its alpha value. Better learners get louder voices.

The weights are then normalized (divided by their sum) so they form a proper probability distribution for the next round.

Key Takeaway: AdaBoost's power comes from exponentially increasing weights on misclassified samples and weighting learner votes by their accuracy, creating a strong classifier from multiple weak ones through mathematical precision.