Implementing AdaBoost with Decision Stumps
You'll learn how to build an AdaBoost classifier in scikit-learn using decision stumps—the simplest possible decision trees—as your weak learners.
What Are Decision Stumps?
A decision stump is a decision tree with only one split—essentially a tree with just a root and two leaves. Think of it as asking a single yes/no question about one feature. Despite being extremely simple (and often barely better than random guessing), these "weak learners" become powerful when combined through AdaBoost's sequential learning process.
Why Decision Stumps for AdaBoost?
AdaBoost shines when you give it simple models that individually don't perform well. Decision stumps are perfect because:
- They're fast to train
- They prevent overfitting at the base learner level
- AdaBoost focuses their collective attention on hard-to-classify examples through reweighting
Implementation in Scikit-learn
In scikit-learn, you create a decision stump by setting max_depth=1 on a DecisionTreeClassifier, then pass it to AdaBoostClassifier:
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
# Create a decision stump (depth-1 tree)
stump = DecisionTreeClassifier(max_depth=1)
# Build AdaBoost with 50 stumps
ada_clf = AdaBoostClassifier(
base_estimator=stump,
n_estimators=50,
learning_rate=1.0
)
# Train and predict as usual
ada_clf.fit(X_train, y_train)
predictions = ada_clf.predict(X_test)
Each of the 50 stumps asks one simple question, but together they form sophisticated decision boundaries by focusing on different parts of the feature space weighted by AdaBoost's reweighting scheme.
Key Takeaway: Decision stumps (depth-1 trees) are ideal weak learners for AdaBoost—simple enough to avoid overfitting individually, yet collectively powerful through adaptive reweighting and sequential combination.