Course contentsShow
← Machine Learning and Deep Learning
Lesson 619 of 3,53815. Neural Network FoundationsFree lesson

Cross-Entropy Mathematics and Information Theory

Understanding the information-theoretic foundation of cross-entropy as KL divergence between target and predicted distributions.

Cross-Entropy Mathematics and Information Theory

You'll learn: Why cross-entropy measures the "distance" between what your model predicts and what's actually true, rooted in information theory.

The Information Theory Foundation

Cross-entropy isn't just a loss function pulled from thin air—it has deep roots in information theory. At its core, it measures how well one probability distribution (your model's predictions) approximates another (the true labels).

Think of it this way: imagine you're trying to communicate messages efficiently. If you know the true distribution of messages, you can design an optimal code. But if you use a code designed for a different distribution (your model's wrong guesses), you'll need more bits on average. That extra cost is the cross-entropy.

The Mathematical Connection

Cross-entropy between true distribution P and predicted distribution Q is defined as:

H(P, Q) = -Σ P(x) log Q(x)

For classification, P is typically a one-hot vector (all zeros except 1.0 for the true class), and Q is your model's predicted probability distribution.

Cross-entropy can also be understood as the sum of two parts:

  • Entropy of the true distribution: H(P) — the inherent uncertainty
  • KL divergence from Q to P: D_KL(P||Q) — how much your prediction differs from truth

So: H(P, Q) = H(P) + D_KL(P||Q)

Since H(P) is constant during training (labels don't change), minimizing cross-entropy is equivalent to minimizing the KL divergence—making your predictions match the true distribution as closely as possible.

Key Takeaway: Cross-entropy loss is information-theoretic: it measures the "coding inefficiency" when using your model's predicted distribution instead of the true distribution, and minimizing it reduces the KL divergence between them.