Course contentsShow
← Machine Learning and Deep Learning
Lesson 3286 of 3,53872. Bias and FairnessFree lesson

Calibration and Calibration Parity

Learn calibration: predicted probabilities match actual outcomes, and calibration parity: requiring this to hold within each protected group.

Calibration and Calibration Parity

What you'll learn: How to ensure predicted probabilities reflect real-world frequencies, both overall and within protected groups.

What is Calibration?

A model is calibrated when its predicted probabilities match actual outcomes. If your model says "70% chance of rain," it should actually rain about 70% of the time across all instances where it made that prediction.

More formally: among all predictions with probability p, approximately p fraction should be positive examples. If you predict 100 people have an 80% chance of loan default, roughly 80 of them should actually default.

Think of it like weather forecasting: a well-calibrated forecaster who says "30% chance of rain" should see rain occur on about 3 out of every 10 days they make that prediction.

Calibration Parity

Calibration parity requires that calibration holds within each protected group. It's not enough for predictions to be calibrated overall—they must be equally well-calibrated for different demographic groups.

For example, if your model predicts 60% recidivism risk:

  • Among predicted Group A members, ~60% should actually recidivate
  • Among predicted Group B members, ~60% should also actually recidivate

This prevents systematic over- or under-estimation of risk for particular groups, which could lead to discriminatory outcomes even if other fairness metrics appear satisfied.

Why It Matters

Calibration parity ensures that decision-makers can trust probability estimates equally across groups. Without it, a "high risk" prediction might mean very different things for different demographics, making fair treatment impossible.

Key Takeaway: Calibration means predicted probabilities match actual outcome frequencies; calibration parity demands this relationship holds separately within each protected group, ensuring probability estimates are equally meaningful across demographics.