Log Loss (Cross-Entropy)
You'll learn why confident wrong predictions are especially dangerous and how log loss measures probability quality.
The Problem with Being Confidently Wrong
Imagine two weather forecasters. The first says "70% chance of rain" when it doesn't rain. The second boldly declares "99% chance of rain" when it doesn't rain. Both were wrong, but the second forecaster was confidently wrong—and that's much worse.
Log loss (also called cross-entropy) penalizes confident wrong predictions far more severely than uncertain wrong predictions. It doesn't just care if you're right or wrong—it cares about how sure you were.
How Log Loss Works
The formula involves the logarithm of your predicted probability:
- If you predict 90% probability for the correct class: small penalty
- If you predict 10% probability for the correct class: huge penalty
- If you predict 1% probability for the correct class: massive penalty
The logarithm creates this exponential punishment curve. Being slightly wrong with high confidence costs you dearly.
Why This Matters for Calibrated Probabilities
Log loss forces your model to output well-calibrated probabilities—predictions that reflect true confidence levels. A model that says "80% positive" should be correct about 80% of the time on those predictions.
This is crucial for:
- Medical diagnosis (need honest uncertainty)
- Risk assessment (financial decisions depend on accurate probabilities)
- Any domain where the probability itself matters, not just the final yes/no decision
Key Takeaway: Log loss heavily penalizes confident wrong predictions, rewarding models that produce honest, well-calibrated probability estimates rather than overconfident guesses.