Calibration Plots and Reliability Diagrams
What You'll Learn
You'll learn how to visually check whether your model's predicted probabilities actually match the real-world frequency of outcomes.
The Core Idea
Imagine your weather app says "80% chance of rain" on ten different days. If it actually rains on about 8 of those days, the app is well-calibrated. If it only rains twice, the app is overconfident—it's not calibrated properly.
A calibration plot (also called a reliability diagram) does exactly this check for your ML model's probability predictions. Here's how it works:
- Bin the predictions: Group all predictions into buckets (e.g., 0-10%, 10-20%, ... 90-100%)
- Calculate actual frequency: For each bucket, count how often the positive class actually occurred
- Plot predicted vs actual: Put predicted probability on the x-axis and observed frequency on the y-axis
Reading the Plot
- Perfect calibration: Points fall on the diagonal line (45-degree line). If your model predicts 70%, the outcome happens 70% of the time.
- Above the line: Your model is underconfident (predicts 30% but happens 50% of the time)
- Below the line: Your model is overconfident (predicts 80% but only happens 60% of the time)
Why This Matters
Many classifiers (like Naive Bayes or uncalibrated neural networks) produce scores that rank examples well but don't represent true probabilities. Before using predictions for decision-making (e.g., medical diagnosis, loan approval), you need trustworthy probabilities—calibration plots reveal whether you have them.
Key Takeaway: Calibration plots visualize whether predicted probabilities align with real-world frequencies by binning predictions and comparing them to actual outcomes—essential for trusting probability-based decisions.