Course contentsShow
← Machine Learning and Deep Learning
Lesson 314 of 3,5386. Tree-Based MethodsFree lesson

Learning Rate and Shrinkage in Boosting

Controlling model complexity and preventing overfitting by scaling tree contributions with a learning rate.

Learning Rate and Shrinkage in Boosting

You'll learn: How to control boosting model complexity and prevent overfitting by scaling each tree's contribution with a learning rate parameter.

The Core Idea

In gradient boosting, each new tree tries to correct the errors of the ensemble built so far. However, letting each tree contribute its full prediction can cause the model to overfit—it learns the training data too well and loses the ability to generalize.

Shrinkage (also called the learning rate) solves this by scaling down each tree's contribution. Instead of adding the full prediction of a new tree, we add only a fraction of it, typically between 0.01 and 0.3.

The Mathematical Adjustment

When building the ensemble sequentially:

F_m(x) = F_(m-1)(x) + tree_m(x)

With shrinkage (learning rate η):

F_m(x) = F_(m-1)(x) + η * tree_m(x)

Where η (eta) is a small positive number, typically 0.1 or less.

The Intuition: Small Steps, Better Path

Think of gradient boosting as hiking down a mountain in fog. Without shrinkage, you take giant leaps based on what you see—you might overshoot the valley or stumble. With a small learning rate, you take tiny, careful steps. It takes longer (more trees), but you're less likely to overshoot and more likely to find the true minimum.

Lower learning rates require more trees but generally produce better-performing models that generalize well to new data.

The Tradeoff

  • Lower learning rate (e.g., 0.01): More trees needed, slower training, better generalization
  • Higher learning rate (e.g., 0.3): Fewer trees needed, faster training, higher overfitting risk

Key Takeaway: The learning rate (shrinkage) parameter scales down each tree's contribution in boosting, trading training speed for better generalization by forcing the model to learn gradually through many small corrections.