Course contentsShow
← Machine Learning and Deep Learning
Lesson 531 of 3,53812. Model Validation and TuningFree lesson

Expected Calibration Error (ECE)

Quantifying calibration quality: computing ECE as weighted average of accuracy-confidence differences across bins.

Expected Calibration Error (ECE)

What You'll Learn

You'll learn how to quantify how well-calibrated your model is by computing a single number that summarizes the gap between predicted confidence and actual accuracy.

The Core Idea

A well-calibrated model should be correct 90% of the time when it's 90% confident, 70% correct when 70% confident, and so on. But how do we measure this alignment in a single metric?

Expected Calibration Error (ECE) gives you that number. Here's how it works:

  1. Bin your predictions: Group predictions by confidence level (e.g., 0-10%, 10-20%, ..., 90-100%)
  2. Measure the gap in each bin: For each bin, calculate the difference between the average predicted confidence and the actual accuracy
  3. Take a weighted average: Compute the overall ECE by averaging these gaps, weighted by how many predictions fell into each bin

The Formula in Plain English

For each confidence bin:

  • Calculate |average_confidence - actual_accuracy|
  • Weight this by the proportion of samples in that bin

Then sum across all bins. Lower ECE = better calibration.

An Analogy

Imagine a weather forecaster who says "70% chance of rain" on 100 days. If it actually rains on exactly 70 of those days, their calibration error for that prediction level is zero. ECE measures this across all confidence levels they use, weighted by how often they make each type of prediction.

Why It Matters

ECE gives you a single score (typically between 0 and 1, or 0% and 100%) that tells you how trustworthy your model's confidence scores are. An ECE of 0.05 means your model's confidence is off by about 5% on average—pretty good! An ECE of 0.25 means serious calibration issues.

Key Takeaway: ECE quantifies calibration quality as a weighted average of accuracy-confidence gaps across bins, giving you one number to judge how trustworthy your model's probabilities are.