Gradient Descent for Logistic Regression
What you'll learn: How to apply gradient descent to train a logistic regression model by minimizing the binary cross-entropy loss.
The Core Idea
You've already learned how gradient descent updates parameters by following the gradient downhill, and you've seen the gradient of the cross-entropy loss function for logistic regression. Now we bring these pieces together to actually train a logistic regression classifier.
Just like with linear regression, gradient descent iteratively adjusts the weights (coefficients) and bias to minimize the loss function. The key difference? We're now minimizing binary cross-entropy instead of mean squared error, and our predictions pass through the sigmoid function.
The Update Process
The gradient descent update rule for logistic regression looks remarkably similar to linear regression:
w = w - α · ∇Loss
Where:
- w represents your weights (coefficients)
- α is the learning rate
- ∇Loss is the gradient of the cross-entropy loss
The beautiful part: even though the loss function is different (cross-entropy vs MSE), the gradient computation yields a clean form. For each training example, the error term is simply (prediction - actual_label), weighted by the feature values.
The Algorithm
- Initialize weights and bias (often to zeros or small random values)
- Compute predictions using the sigmoid function
- Calculate the cross-entropy loss
- Compute gradients of the loss with respect to each parameter
- Update parameters using the gradient descent rule
- Repeat until convergence
You can use batch, stochastic, or mini-batch gradient descent, just like with linear regression.
Key Takeaway: Gradient descent for logistic regression minimizes binary cross-entropy by iteratively updating weights in the direction that reduces prediction error, enabling the model to learn optimal decision boundaries for classification.