Course contentsShow
← Machine Learning and Deep Learning
Lesson 489 of 3,53811. Model Evaluation MetricsFree lesson

Calibration Plots and Reliability Diagrams

Visualizing how predicted probabilities align with observed frequencies using binned calibration curves.

Calibration Plots and Reliability Diagrams

What You'll Learn

You'll learn how to visually check whether your model's predicted probabilities actually match the real-world frequency of outcomes.

The Core Idea

Imagine your weather app says "80% chance of rain" on ten different days. If it actually rains on about 8 of those days, the app is well-calibrated. If it only rains twice, the app is overconfident—it's not calibrated properly.

A calibration plot (also called a reliability diagram) does exactly this check for your ML model's probability predictions. Here's how it works:

  1. Bin the predictions: Group all predictions into buckets (e.g., 0-10%, 10-20%, ... 90-100%)
  2. Calculate actual frequency: For each bucket, count how often the positive class actually occurred
  3. Plot predicted vs actual: Put predicted probability on the x-axis and observed frequency on the y-axis

Reading the Plot

  • Perfect calibration: Points fall on the diagonal line (45-degree line). If your model predicts 70%, the outcome happens 70% of the time.
  • Above the line: Your model is underconfident (predicts 30% but happens 50% of the time)
  • Below the line: Your model is overconfident (predicts 80% but only happens 60% of the time)

Why This Matters

Many classifiers (like Naive Bayes or uncalibrated neural networks) produce scores that rank examples well but don't represent true probabilities. Before using predictions for decision-making (e.g., medical diagnosis, loan approval), you need trustworthy probabilities—calibration plots reveal whether you have them.

Key Takeaway: Calibration plots visualize whether predicted probabilities align with real-world frequencies by binning predictions and comparing them to actual outcomes—essential for trusting probability-based decisions.