Course contentsShow
← Machine Learning and Deep Learning
Lesson 264 of 3,5384. Classification with Logistic RegressionFree lesson

Cross-Entropy Loss for Multiclass

Deriving the categorical cross-entropy loss function that measures disagreement between predicted probabilities and true class labels.

Cross-Entropy Loss for Multiclass

What you'll learn: How to measure how wrong your model's predicted class probabilities are across multiple classes using a single loss value.

The Scoring Problem

Remember from binary cross-entropy that we needed a way to measure how badly our model's probability predictions disagreed with the actual labels. With multiclass classification, you're now dealing with multiple probability predictions per sample—one for each class—but you still need one number that tells you how wrong you are overall.

The Core Idea

Cross-entropy loss (also called categorical cross-entropy) compares your model's predicted probability distribution (the softmax output) with the true distribution (the actual class).

Here's the formula:

L = -Σ y_i * log(ŷ_i)

Where:

  • y_i is the true label (1 for the correct class, 0 for all others—a one-hot encoded vector)
  • ŷ_i is your model's predicted probability for class i
  • The sum runs over all classes

Intuition: If the true class is class 2, then y_2 = 1 and all other y_i = 0. The loss simplifies to -log(ŷ_2). So you're only "penalized" based on the probability you assigned to the correct class. If you predicted 0.9 for the right class, -log(0.9) ≈ 0.105 (low loss). If you predicted 0.1, -log(0.1) ≈ 2.3 (high loss).

Why It Works

Cross-entropy comes from information theory and maximum likelihood estimation. It naturally pushes the model to assign high probability to the correct class and distributes remaining probability across wrong classes.

Key Takeaway: Cross-entropy loss for multiclass classification measures the disagreement between predicted class probabilities and true labels by heavily penalizing low confidence in the correct class.