Course contentsShow
← Machine Learning and Deep Learning
Lesson 252 of 3,5384. Classification with Logistic RegressionFree lesson

Gradient Descent for Logistic Regression

Applying gradient descent to minimize cross-entropy and train logistic regression models.

Gradient Descent for Logistic Regression

What you'll learn: How to apply gradient descent to train a logistic regression model by minimizing the binary cross-entropy loss.

The Core Idea

You've already learned how gradient descent updates parameters by following the gradient downhill, and you've seen the gradient of the cross-entropy loss function for logistic regression. Now we bring these pieces together to actually train a logistic regression classifier.

Just like with linear regression, gradient descent iteratively adjusts the weights (coefficients) and bias to minimize the loss function. The key difference? We're now minimizing binary cross-entropy instead of mean squared error, and our predictions pass through the sigmoid function.

The Update Process

The gradient descent update rule for logistic regression looks remarkably similar to linear regression:

w = w - α · ∇Loss

Where:

  • w represents your weights (coefficients)
  • α is the learning rate
  • ∇Loss is the gradient of the cross-entropy loss

The beautiful part: even though the loss function is different (cross-entropy vs MSE), the gradient computation yields a clean form. For each training example, the error term is simply (prediction - actual_label), weighted by the feature values.

The Algorithm

  1. Initialize weights and bias (often to zeros or small random values)
  2. Compute predictions using the sigmoid function
  3. Calculate the cross-entropy loss
  4. Compute gradients of the loss with respect to each parameter
  5. Update parameters using the gradient descent rule
  6. Repeat until convergence

You can use batch, stochastic, or mini-batch gradient descent, just like with linear regression.

Key Takeaway: Gradient descent for logistic regression minimizes binary cross-entropy by iteratively updating weights in the direction that reduces prediction error, enabling the model to learn optimal decision boundaries for classification.