Course contentsShow
← Machine Learning and Deep Learning
Lesson 530 of 3,53812. Model Validation and TuningFree lesson

Reliability Diagrams

Visualizing calibration with reliability plots: binning predictions and comparing predicted vs observed frequencies.

Reliability Diagrams

What you'll learn: How to visually check if your model's predicted probabilities match real-world outcomes using reliability diagrams.

What Are Reliability Diagrams?

A reliability diagram (also called a calibration plot) shows whether your model's probability predictions are trustworthy. When your model says "80% chance of class A," do 80% of those predictions actually turn out to be class A in reality?

The diagram works by:

  1. Binning predictions — grouping all predictions into buckets (e.g., 0-10%, 10-20%, ..., 90-100%)
  2. Calculating observed frequency — for each bin, counting how many instances actually belonged to the positive class
  3. Plotting predicted vs observed — comparing what the model predicted against what really happened

How to Read It

  • X-axis: Mean predicted probability in each bin
  • Y-axis: Actual fraction of positives observed in that bin
  • Perfect calibration: Points lie on the diagonal line (predicted = observed)
  • Below the line: Model is overconfident (predicts higher probabilities than observed)
  • Above the line: Model is underconfident (predicts lower probabilities than observed)

Why It Matters

Imagine a medical diagnosis model that says "90% cancer risk" but only 50% of those patients actually have cancer. That's dangerous overconfidence! A reliability diagram reveals this mismatch at a glance, helping you decide whether to trust the probabilities or apply calibration techniques.

Key Takeaway: Reliability diagrams visualize calibration by binning predictions and plotting predicted probabilities against actual observed frequencies—a perfectly calibrated model's points fall on the diagonal line.