Course contentsShow
← Machine Learning and Deep Learning
Lesson 490 of 3,53811. Model Evaluation MetricsFree lesson

Expected Calibration Error (ECE)

Quantifying calibration by measuring weighted average difference between confidence and accuracy across bins.

Expected Calibration Error (ECE)

You'll learn: How to quantify a model's calibration quality with a single number by measuring how well predicted probabilities match actual outcomes.

What Is ECE?

Remember calibration plots from the previous lesson? They show us visually whether a model's confidence matches reality. Expected Calibration Error (ECE) turns that visual assessment into a concrete metric you can track and compare.

ECE measures the weighted average difference between a model's confidence (predicted probabilities) and its accuracy (actual correctness) across multiple bins.

How It Works

Imagine you're grading a weather forecaster who says "70% chance of rain" on many days:

  1. Group predictions into bins: Collect all predictions between 60-80% confidence into one bucket, 80-100% into another, etc.
  2. Measure accuracy per bin: In the 60-80% bin, did it actually rain 70% of the time?
  3. Calculate differences: For each bin, find |confidence - accuracy|
  4. Weight by bin size: Bins with more predictions matter more
  5. Average everything: Sum up the weighted errors

The formula captures this as:

ECE = Σ (number_in_bin / total) × |avg_confidence_in_bin - accuracy_in_bin|

Why This Matters

A perfectly calibrated model has ECE = 0. When your model predicts 80% confidence, it should be correct exactly 80% of the time. Higher ECE means your model is overconfident or underconfident—not just wrong in its predictions, but wrong about how sure it is.

This is critical in healthcare, finance, or autonomous driving where "how confident should we be?" matters as much as "what's the prediction?"

Key Takeaway: ECE distills calibration quality into one number: the average gap between predicted confidence and actual accuracy, weighted by how many predictions fall into each confidence range.