Gradient Descent for Logistic Regression
What you'll learn: How to apply gradient descent to optimize logistic regression by minimizing cross-entropy loss and finding the best model parameters.
The Core Idea
You've already learned gradient descent for linear regression and defined cross-entropy loss for logistic regression. Now we bring them together. Just as we used gradients to update weights in linear regression, we'll compute gradients of the cross-entropy loss with respect to our logistic regression parameters and iteratively adjust them to minimize prediction error.
Computing the Gradient
For logistic regression with hypothesis h(x) = σ(θᵀx) (where σ is the sigmoid function) and cross-entropy loss, the gradient turns out to have a remarkably clean form:
∂L/∂θⱼ = (1/m) Σᵢ (h(xⁱ) - yⁱ) · xⱼⁱ
This looks almost identical to linear regression's gradient! The difference: h(x) now outputs probabilities via the sigmoid function rather than raw predictions.
The Update Rule
Using this gradient, we update each parameter using the familiar pattern:
θⱼ := θⱼ - α · ∂L/∂θⱼ
Where α is your learning rate. Each iteration moves the parameters in the direction that reduces cross-entropy loss, gradually improving how well your model separates the two classes.
Why This Works
Think of it as navigating a probability landscape. Cross-entropy measures how "surprised" your model is by the true labels. Gradient descent finds the θ values that make your model least surprised—maximizing confidence in correct predictions while pushing decision boundaries to the right places.
Key Takeaway: Gradient descent for logistic regression minimizes cross-entropy loss using gradients that have the same form as linear regression but work through the sigmoid function, iteratively finding parameters that best separate classes based on probability estimates.