Learning Rate and Shrinkage in Boosting
You'll learn: How to control boosting model complexity and prevent overfitting by scaling each tree's contribution with a learning rate parameter.
The Core Idea
In gradient boosting, each new tree tries to correct the errors of the ensemble built so far. However, letting each tree contribute its full prediction can cause the model to overfit—it learns the training data too well and loses the ability to generalize.
Shrinkage (also called the learning rate) solves this by scaling down each tree's contribution. Instead of adding the full prediction of a new tree, we add only a fraction of it, typically between 0.01 and 0.3.
The Mathematical Adjustment
When building the ensemble sequentially:
F_m(x) = F_(m-1)(x) + tree_m(x)
With shrinkage (learning rate η):
F_m(x) = F_(m-1)(x) + η * tree_m(x)
Where η (eta) is a small positive number, typically 0.1 or less.
The Intuition: Small Steps, Better Path
Think of gradient boosting as hiking down a mountain in fog. Without shrinkage, you take giant leaps based on what you see—you might overshoot the valley or stumble. With a small learning rate, you take tiny, careful steps. It takes longer (more trees), but you're less likely to overshoot and more likely to find the true minimum.
Lower learning rates require more trees but generally produce better-performing models that generalize well to new data.
The Tradeoff
- Lower learning rate (e.g., 0.01): More trees needed, slower training, better generalization
- Higher learning rate (e.g., 0.3): Fewer trees needed, faster training, higher overfitting risk
Key Takeaway: The learning rate (shrinkage) parameter scales down each tree's contribution in boosting, trading training speed for better generalization by forcing the model to learn gradually through many small corrections.