Expected Calibration Error (ECE)
What You'll Learn
You'll learn how to quantify how well-calibrated your model is by computing a single number that summarizes the gap between predicted confidence and actual accuracy.
The Core Idea
A well-calibrated model should be correct 90% of the time when it's 90% confident, 70% correct when 70% confident, and so on. But how do we measure this alignment in a single metric?
Expected Calibration Error (ECE) gives you that number. Here's how it works:
- Bin your predictions: Group predictions by confidence level (e.g., 0-10%, 10-20%, ..., 90-100%)
- Measure the gap in each bin: For each bin, calculate the difference between the average predicted confidence and the actual accuracy
- Take a weighted average: Compute the overall ECE by averaging these gaps, weighted by how many predictions fell into each bin
The Formula in Plain English
For each confidence bin:
- Calculate |average_confidence - actual_accuracy|
- Weight this by the proportion of samples in that bin
Then sum across all bins. Lower ECE = better calibration.
An Analogy
Imagine a weather forecaster who says "70% chance of rain" on 100 days. If it actually rains on exactly 70 of those days, their calibration error for that prediction level is zero. ECE measures this across all confidence levels they use, weighted by how often they make each type of prediction.
Why It Matters
ECE gives you a single score (typically between 0 and 1, or 0% and 100%) that tells you how trustworthy your model's confidence scores are. An ECE of 0.05 means your model's confidence is off by about 5% on average—pretty good! An ECE of 0.25 means serious calibration issues.
Key Takeaway: ECE quantifies calibration quality as a weighted average of accuracy-confidence gaps across bins, giving you one number to judge how trustworthy your model's probabilities are.