What is Model Calibration?
You'll learn: How to tell if your model's predicted probabilities actually mean what they say they mean.
The Core Idea
When your classification model predicts "80% probability of rain," does it really rain about 80% of the time in those situations? Model calibration answers this question.
A well-calibrated model produces predicted probabilities that match the true frequencies of outcomes. If you collect all predictions where your model said "70% probability of Class A," and Class A actually occurred about 70% of the time, your model is calibrated.
Many models (like decision trees or neural networks) can be very accurate but poorly calibrated—they might predict extreme probabilities (like 0.95 or 0.05) too often, even when the true uncertainty is higher.
Why Calibration Matters for Decision-Making
Imagine a medical diagnosis system that predicts disease probability:
- Uncalibrated: Says "90% chance of disease" but the patient actually has disease only 60% of the time
- Calibrated: Says "90% chance" and disease truly occurs ~90% of the time
The calibrated model lets doctors make informed decisions about treatment risks versus benefits. The uncalibrated model might lead to unnecessary interventions or missed cases.
Calibration is crucial whenever you need to:
- Make cost-benefit decisions based on probabilities
- Set decision thresholds appropriately
- Communicate uncertainty to stakeholders
- Combine model outputs with other information sources
How We Check Calibration
You've already learned about Brier Score and Calibration Plots (reliability diagrams)—these tools help us visualize and quantify whether predicted probabilities align with observed frequencies across different probability ranges.
Key Takeaway: Model calibration ensures that predicted probabilities reflect true outcome frequencies, enabling trustworthy probability-based decision-making rather than just class predictions.