Ensemble Selection and Hyperparameters
You'll learn when to choose bagging, boosting, or stacking, and which hyperparameters matter most for ensemble performance.
Choosing the Right Ensemble Strategy
Each ensemble method addresses different problems:
Bagging (like Random Forests) works best when you have:
- High-variance models that overfit easily (like deep decision trees)
- Independent features where randomness helps
- Need for parallel training and fast predictions
- Goal: reduce variance without increasing bias
Boosting (like AdaBoost, Gradient Boosting, XGBoost) excels when you have:
- High-bias models that underfit (like shallow trees)
- Complex patterns requiring sequential refinement
- Tolerance for longer training times
- Goal: reduce both bias and variance through iterative correction
Stacking shines when you have:
- Multiple diverse models with different strengths
- Sufficient data to train a meta-learner
- Need for maximum predictive power (competitions)
- Goal: combine complementary model perspectives
Critical Hyperparameters
Number of estimators (n_estimators): How many base models to train
- Bagging: More is usually better (plateaus eventually)
- Boosting: Too many causes overfitting; requires validation
Learning rate (boosting only): How much each model contributes
- Lower rates (0.01-0.1) with more estimators often beat high rates
- Trade-off: training time vs model quality
Max depth/complexity of base models:
- Bagging: deeper trees okay (averaging reduces variance)
- Boosting: shallow trees (2-8 levels) prevent overfitting
Think of it like building a team: bagging assembles independent experts working in parallel, boosting coaches a team that learns from past mistakes sequentially, and stacking appoints specialists with a manager who combines their advice.
Key Takeaway: Use bagging to tame overfitting models, boosting to strengthen weak models iteratively, and stacking to merge diverse approaches—then tune n_estimators and learning_rate based on validation performance.