Course contentsShow
← Machine Learning and Deep Learning
Lesson 251 of 3,5384. Classification with Logistic RegressionFree lesson

Gradient of the Loss Function

Deriving the gradient of binary cross-entropy with respect to model parameters.

Gradient of the Loss Function

What you'll learn: How to compute the gradient that tells us how to adjust logistic regression parameters to minimize binary cross-entropy loss.


The Core Idea

You've learned that binary cross-entropy measures how wrong your logistic regression predictions are. Now you need to know which direction to adjust your model parameters (weights and bias) to make better predictions. The gradient provides exactly this information—it's the mathematical "arrow" pointing toward the steepest increase in loss, so you'll move in the opposite direction to minimize it.

Deriving the Gradient

For logistic regression with sigmoid function σ(z) = 1/(1 + e^(-z)) and z = wx + b, the gradient with respect to weights turns out to be surprisingly elegant:

∂Loss/∂w = (σ(wx + b) - y) · x

And for the bias:

∂Loss/∂b = σ(wx + b) - y

Notice the beautiful pattern: the gradient is simply the prediction error (prediction - actual) multiplied by the input features (for weights).

Why This Matters

This gradient tells you:

  • Magnitude: How much to adjust parameters (larger error = larger adjustment)
  • Direction: Whether to increase or decrease parameters (sign of the error)
  • Feature scaling: Features with larger values contribute more to weight updates

When your prediction σ(wx + b) is close to the true label y, the gradient is small—you're nearly there. When the prediction is way off, the gradient is large—you need bigger corrections.

This gradient is what gradient descent uses in each update step to iteratively improve your logistic regression model.

Key Takeaway: The gradient of binary cross-entropy with respect to model parameters is simply the prediction error multiplied by the input features—this elegant formula guides how to adjust weights and bias to minimize classification loss.