Calibration and Calibration Parity
What you'll learn: How to ensure predicted probabilities reflect real-world frequencies, both overall and within protected groups.
What is Calibration?
A model is calibrated when its predicted probabilities match actual outcomes. If your model says "70% chance of rain," it should actually rain about 70% of the time across all instances where it made that prediction.
More formally: among all predictions with probability p, approximately p fraction should be positive examples. If you predict 100 people have an 80% chance of loan default, roughly 80 of them should actually default.
Think of it like weather forecasting: a well-calibrated forecaster who says "30% chance of rain" should see rain occur on about 3 out of every 10 days they make that prediction.
Calibration Parity
Calibration parity requires that calibration holds within each protected group. It's not enough for predictions to be calibrated overall—they must be equally well-calibrated for different demographic groups.
For example, if your model predicts 60% recidivism risk:
- Among predicted Group A members, ~60% should actually recidivate
- Among predicted Group B members, ~60% should also actually recidivate
This prevents systematic over- or under-estimation of risk for particular groups, which could lead to discriminatory outcomes even if other fairness metrics appear satisfied.
Why It Matters
Calibration parity ensures that decision-makers can trust probability estimates equally across groups. Without it, a "high risk" prediction might mean very different things for different demographics, making fair treatment impossible.
Key Takeaway: Calibration means predicted probabilities match actual outcome frequencies; calibration parity demands this relationship holds separately within each protected group, ensuring probability estimates are equally meaningful across demographics.