Course contentsShow
← Machine Learning and Deep Learning
Lesson 313 of 3,5386. Tree-Based MethodsFree lesson

Gradient Boosting for Classification

Extending gradient boosting to classification tasks using log-loss and probability predictions.

Gradient Boosting for Classification

What you'll learn: How gradient boosting adapts from regression to classification by predicting probabilities and optimizing log-loss.

The Core Idea

You've seen gradient boosting build regression models by iteratively correcting residual errors. Classification works similarly, but instead of predicting continuous values directly, we predict probabilities and use a different loss function.

For binary classification, gradient boosting:

  1. Starts with an initial probability estimate (often the log-odds of the positive class proportion)
  2. Computes gradients of the log-loss (also called binary cross-entropy) with respect to current predictions
  3. Fits a tree to these gradients, just like fitting to residuals in regression
  4. Updates probability predictions by adding the tree's output, scaled by a learning rate
  5. Repeats for many iterations

From Regression to Classification

Think of it this way: instead of asking "how far off am I?" (regression residuals), you're asking "how wrong is my probability estimate?" The gradient of log-loss tells you which direction to adjust probabilities for each training sample.

The log-loss heavily penalizes confident wrong predictions—if you predict 95% probability of class 1 but the true label is 0, the gradient pushes hard to correct that mistake.

After building all trees, you convert raw predictions back to probabilities using the sigmoid function (for binary classification), giving you the final class probabilities.

Why Log-Loss?

Log-loss is the natural choice because it directly measures how well predicted probabilities match true labels. Minimizing log-loss maximizes the likelihood of observing the correct classes.

Key Takeaway: Gradient boosting extends to classification by predicting probabilities through iterative gradient descent on log-loss, using the same tree-building framework as regression but optimizing a probability-appropriate objective.