Course contentsShow
← Computer Science
Lesson 2831 of 2,87263. Classical ML AlgorithmsFree lesson

Gradient Descent for Logistic Regression

Derive and apply gradient descent to minimize cross-entropy loss and find optimal parameters.

Gradient Descent for Logistic Regression

What you'll learn: How to apply gradient descent to optimize logistic regression by minimizing cross-entropy loss and finding the best model parameters.

The Core Idea

You've already learned gradient descent for linear regression and defined cross-entropy loss for logistic regression. Now we bring them together. Just as we used gradients to update weights in linear regression, we'll compute gradients of the cross-entropy loss with respect to our logistic regression parameters and iteratively adjust them to minimize prediction error.

Computing the Gradient

For logistic regression with hypothesis h(x) = σ(θᵀx) (where σ is the sigmoid function) and cross-entropy loss, the gradient turns out to have a remarkably clean form:

∂L/∂θⱼ = (1/m) Σᵢ (h(xⁱ) - yⁱ) · xⱼⁱ

This looks almost identical to linear regression's gradient! The difference: h(x) now outputs probabilities via the sigmoid function rather than raw predictions.

The Update Rule

Using this gradient, we update each parameter using the familiar pattern:

θⱼ := θⱼ - α · ∂L/∂θⱼ

Where α is your learning rate. Each iteration moves the parameters in the direction that reduces cross-entropy loss, gradually improving how well your model separates the two classes.

Why This Works

Think of it as navigating a probability landscape. Cross-entropy measures how "surprised" your model is by the true labels. Gradient descent finds the θ values that make your model least surprised—maximizing confidence in correct predictions while pushing decision boundaries to the right places.

Key Takeaway: Gradient descent for logistic regression minimizes cross-entropy loss using gradients that have the same form as linear regression but work through the sigmoid function, iteratively finding parameters that best separate classes based on probability estimates.