Course contentsShow
← Computer Science
Lesson 2834 of 2,87263. Classical ML AlgorithmsFree lesson

Multiclass Logistic Regression

Extend binary logistic regression to multiple classes using one-vs-rest and softmax approaches.

Multiclass Logistic Regression

What you'll learn: How to extend binary logistic regression to handle classification problems with three or more classes.

Binary logistic regression uses the sigmoid function to output probabilities for two classes (0 or 1). But what if you need to classify images into "cat," "dog," or "bird"? You need multiclass classification.

Two Main Approaches

One-vs-Rest (OvR)

Also called "one-vs-all," this strategy trains K separate binary classifiers for K classes. Each classifier answers: "Is this example class i or not?" For a new input, you run all K classifiers and choose the class with the highest probability output.

Analogy: Imagine K expert judges, each specialized in recognizing one animal type. Each judge gives a confidence score, and you trust the most confident one.

Softmax Regression

Instead of K independent models, softmax uses a single model that outputs K probabilities simultaneously. It replaces the sigmoid function with the softmax function, which ensures all class probabilities sum to 1:

P(y = k | x) = exp(w_k · x) / Σ exp(w_j · x)

where w_k is the weight vector for class k.

The softmax naturally handles multiple classes and is trained using cross-entropy loss across all classes. Gradient descent updates all weight vectors simultaneously based on the prediction errors.

Which to Use?

One-vs-Rest is simpler to implement (reuse binary classifiers) but treats classes independently. Softmax is more elegant mathematically, directly models class probabilities, and often performs better when classes are mutually exclusive. Both can incorporate L1 or L2 regularization just like binary logistic regression.

Key Takeaway: Multiclass logistic regression extends binary classification through one-vs-rest (training multiple binary classifiers) or softmax (a unified probabilistic model outputting all class probabilities at once).