Categorical Cross-Entropy Loss
What you'll learn: How to measure prediction error for multi-class classification problems where each example belongs to exactly one of several possible classes.
The Multi-Class Extension
You've already learned about Binary Cross-Entropy Loss for two-class problems. Categorical Cross-Entropy is its natural extension to multiple classes (3 or more). Instead of asking "Is this A or B?", we're now asking "Is this A, B, C, D, or E?"
How It Works
Categorical cross-entropy compares two probability distributions:
- The true label (one-hot encoded): A vector like
[0, 1, 0, 0]meaning "definitely class 2" - The predicted probabilities (from softmax): A vector like
[0.1, 0.7, 0.1, 0.1]representing confidence across all classes
The loss formula is:
Loss = -Σ(y_true × log(y_pred))
where the sum runs over all classes. Since the true label is one-hot encoded (all zeros except one 1), only the predicted probability of the correct class contributes to the loss.
Analogy
Imagine a teacher grading a multiple-choice test where students must show confidence levels for each answer. A student who puts 90% confidence on the correct answer gets penalized less than one who puts only 20% on it—even if both technically "chose" the right answer.
Why Softmax Outputs?
Categorical cross-entropy expects the network's output layer to use a softmax activation, which ensures predictions are valid probabilities (positive and sum to 1). The one-hot encoded target guarantees exactly one class is correct per example.
Key Takeaway: Categorical cross-entropy measures how well predicted class probabilities match one-hot encoded true labels, making it the standard loss function for multi-class classification with mutually exclusive classes.