Course contentsShow
← Machine Learning and Deep Learning
Lesson 532 of 3,53812. Model Validation and TuningFree lesson

Why Models Become Miscalibrated

Understanding sources of miscalibration: overconfidence in neural networks, class imbalance, and overfitting effects.

Why Models Become Miscalibrated

What you'll learn: You'll discover the common reasons why machine learning models produce overly confident or poorly calibrated probability predictions.

The Core Problem

Even accurate models can be badly calibrated—they might predict the right class but assign wildly incorrect confidence scores. A model saying "95% confident this is a cat" when historically only 60% of such predictions are correct is miscalibrated. Understanding why this happens helps you fix it.

Three Major Sources of Miscalibration

Overconfidence in Neural Networks
Deep neural networks are notorious for outputting extreme probabilities (near 0 or 1) even when uncertain. This happens because networks optimize for accuracy, not probability accuracy. The softmax function can amplify small differences in logits into very confident predictions, especially with many parameters and flexible architectures.

Class Imbalance
When one class dominates your dataset (say, 95% negative, 5% positive), models learn to favor the majority class. They may predict rare classes with inappropriately low confidence or fail to adjust their thresholds properly. The base rate skew distorts the relationship between predicted probabilities and true frequencies.

Overfitting Effects
An overfitted model memorizes training patterns and becomes overconfident on training data. This false confidence doesn't transfer to validation data—the model hasn't learned generalizable uncertainty. Regularization helps, but complex models can still produce miscalibrated predictions even with decent accuracy.

Think of it like a student who memorizes answers without understanding concepts: they'll answer confidently but be wrong more often than their confidence suggests.

Key Takeaway: Models become miscalibrated primarily through neural network overconfidence, class imbalance distorting probability thresholds, and overfitting creating false certainty—understanding these sources is the first step toward producing trustworthy probability estimates.