Sparse Categorical Cross-Entropy
What You'll Learn
You'll learn how to use integer class labels directly instead of one-hot vectors when training multi-class neural networks, making your code more memory-efficient.
The Problem with Standard Categorical Cross-Entropy
In the previous lesson, you learned about Categorical Cross-Entropy Loss, which expects your target labels as one-hot encoded vectors. For example, if you have 1,000 classes, each label is a vector of 1,000 elements with a single 1 and 999 zeros:
Class 3 → [0, 0, 0, 1, 0, 0, ..., 0] # 1000 elements!
This works, but it's wasteful. Why store 1,000 numbers when you could just store the integer 3?
Enter Sparse Categorical Cross-Entropy
Sparse Categorical Cross-Entropy computes exactly the same loss value as regular categorical cross-entropy, but it accepts integer labels directly:
Class 3 → 3 # Just the integer!
The math is identical—it still measures how different your predicted probability distribution is from the true class. The "sparse" refers to avoiding the sparse (mostly zeros) one-hot representation.
When to Use It
Use Sparse Categorical Cross-Entropy when:
- You have many classes (e.g., 1,000+ categories)
- Memory efficiency matters
- Your labels are naturally integers (like class IDs)
Analogy
Think of it like addressing mail. One-hot encoding is writing "not house 1, not house 2, not house 3, YES house 4, not house 5..." Sparse encoding simply writes "house 4." Both deliver to the same place, but one is much more efficient!
Key Takeaway: Sparse Categorical Cross-Entropy uses integer labels instead of one-hot vectors to achieve the same multi-class classification loss with dramatically better memory efficiency, especially valuable when dealing with many classes.