Gradient Descent for Softmax Regression
What you'll learn: How to compute gradients for multiclass models and update their parameters to minimize cross-entropy loss.
The Core Idea
Just like you used gradient descent to optimize logistic regression, you'll now apply the same principle to softmax regression—but with a key difference: you're updating weight matrices instead of single weight vectors, because you have one set of weights per class.
In softmax regression, you have a weight matrix W where each column (or row, depending on convention) corresponds to one class. When you compute the gradient of the cross-entropy loss with respect to W, you get a matrix of the same shape showing how to adjust each weight.
How It Works
The gradient descent update rule follows the familiar pattern:
W_new = W_old - learning_rate × gradient
The elegant part: the gradient for softmax with cross-entropy has a beautifully simple form. For each training example, it's essentially:
(predicted_probabilities - true_label_one_hot) × input_features
This means if your model predicts class probabilities [0.7, 0.2, 0.1] but the true label is class 2 (one-hot: [0, 0, 1]), the error is [0.7, 0.2, -0.9]. You multiply this error by your input features to get the gradient direction for updating weights.
You can use batch, stochastic, or mini-batch gradient descent, just like with binary logistic regression. The mathematics scale naturally to handle multiple classes simultaneously.
Key Takeaway: Gradient descent for softmax regression updates a weight matrix by computing how predicted class probabilities differ from true labels, scaled by input features—enabling your model to learn multiclass decision boundaries iteratively.