Course contentsShow
← Machine Learning and Deep Learning
Lesson 617 of 3,53815. Neural Network FoundationsFree lesson

Categorical Cross-Entropy Loss

Learning the multi-class generalization of cross-entropy, its use with softmax outputs, and one-hot encoded targets.

Categorical Cross-Entropy Loss

What you'll learn: How to measure prediction error for multi-class classification problems where each example belongs to exactly one of several possible classes.

The Multi-Class Extension

You've already learned about Binary Cross-Entropy Loss for two-class problems. Categorical Cross-Entropy is its natural extension to multiple classes (3 or more). Instead of asking "Is this A or B?", we're now asking "Is this A, B, C, D, or E?"

How It Works

Categorical cross-entropy compares two probability distributions:

  1. The true label (one-hot encoded): A vector like [0, 1, 0, 0] meaning "definitely class 2"
  2. The predicted probabilities (from softmax): A vector like [0.1, 0.7, 0.1, 0.1] representing confidence across all classes

The loss formula is:

Loss = -Σ(y_true × log(y_pred))

where the sum runs over all classes. Since the true label is one-hot encoded (all zeros except one 1), only the predicted probability of the correct class contributes to the loss.

Analogy

Imagine a teacher grading a multiple-choice test where students must show confidence levels for each answer. A student who puts 90% confidence on the correct answer gets penalized less than one who puts only 20% on it—even if both technically "chose" the right answer.

Why Softmax Outputs?

Categorical cross-entropy expects the network's output layer to use a softmax activation, which ensures predictions are valid probabilities (positive and sum to 1). The one-hot encoded target guarantees exactly one class is correct per example.

Key Takeaway: Categorical cross-entropy measures how well predicted class probabilities match one-hot encoded true labels, making it the standard loss function for multi-class classification with mutually exclusive classes.