Reliability Diagrams
What you'll learn: How to visually check if your model's predicted probabilities match real-world outcomes using reliability diagrams.
What Are Reliability Diagrams?
A reliability diagram (also called a calibration plot) shows whether your model's probability predictions are trustworthy. When your model says "80% chance of class A," do 80% of those predictions actually turn out to be class A in reality?
The diagram works by:
- Binning predictions — grouping all predictions into buckets (e.g., 0-10%, 10-20%, ..., 90-100%)
- Calculating observed frequency — for each bin, counting how many instances actually belonged to the positive class
- Plotting predicted vs observed — comparing what the model predicted against what really happened
How to Read It
- X-axis: Mean predicted probability in each bin
- Y-axis: Actual fraction of positives observed in that bin
- Perfect calibration: Points lie on the diagonal line (predicted = observed)
- Below the line: Model is overconfident (predicts higher probabilities than observed)
- Above the line: Model is underconfident (predicts lower probabilities than observed)
Why It Matters
Imagine a medical diagnosis model that says "90% cancer risk" but only 50% of those patients actually have cancer. That's dangerous overconfidence! A reliability diagram reveals this mismatch at a glance, helping you decide whether to trust the probabilities or apply calibration techniques.
Key Takeaway: Reliability diagrams visualize calibration by binning predictions and plotting predicted probabilities against actual observed frequencies—a perfectly calibrated model's points fall on the diagonal line.