Cross-Entropy Loss for Multiclass
What you'll learn: How to measure how wrong your model's predicted class probabilities are across multiple classes using a single loss value.
The Scoring Problem
Remember from binary cross-entropy that we needed a way to measure how badly our model's probability predictions disagreed with the actual labels. With multiclass classification, you're now dealing with multiple probability predictions per sample—one for each class—but you still need one number that tells you how wrong you are overall.
The Core Idea
Cross-entropy loss (also called categorical cross-entropy) compares your model's predicted probability distribution (the softmax output) with the true distribution (the actual class).
Here's the formula:
L = -Σ y_i * log(ŷ_i)
Where:
y_iis the true label (1 for the correct class, 0 for all others—a one-hot encoded vector)ŷ_iis your model's predicted probability for classi- The sum runs over all classes
Intuition: If the true class is class 2, then y_2 = 1 and all other y_i = 0. The loss simplifies to -log(ŷ_2). So you're only "penalized" based on the probability you assigned to the correct class. If you predicted 0.9 for the right class, -log(0.9) ≈ 0.105 (low loss). If you predicted 0.1, -log(0.1) ≈ 2.3 (high loss).
Why It Works
Cross-entropy comes from information theory and maximum likelihood estimation. It naturally pushes the model to assign high probability to the correct class and distributes remaining probability across wrong classes.
Key Takeaway: Cross-entropy loss for multiclass classification measures the disagreement between predicted class probabilities and true labels by heavily penalizing low confidence in the correct class.