Calibration Across Groups
What you'll learn: How to adjust model probability outputs so they're equally trustworthy across different demographic groups.
The Core Problem
You've learned that calibration means "when my model says 70%, it should be right 70% of the time." But what if your model is well-calibrated overall yet poorly calibrated within groups? For example, when predicting loan default risk, your model might say "70% risk" for both Group A and Group B applicants, but historically only 50% of Group A defaults while 85% of Group B defaults. The probabilities mean different things for different groups—that's calibration bias.
Post-Processing for Group Calibration
Calibration across groups ensures that predicted probabilities are equally reliable within each demographic subgroup. After training your model, you examine calibration curves separately for each protected group (e.g., by race, gender, age). If they differ, you apply group-specific transformations to the probability scores.
Common Approaches
- Platt scaling per group: Fit a separate logistic regression from raw scores to true labels for each demographic group
- Isotonic regression per group: Use monotonic piecewise-constant functions to map scores to calibrated probabilities
- Temperature scaling variants: Apply group-specific temperature parameters to soften/sharpen probabilities
The key is you're not retraining the model—you're only adjusting its output probabilities based on observed calibration patterns in your validation set, separately for each group.
Why It Matters
Miscalibrated probabilities mislead decision-makers differently across groups. In healthcare, overconfident predictions for one demographic could lead to under-treatment; in lending, it affects who gets fair credit assessments.
Key Takeaway: Calibration across groups post-processes model scores to ensure predicted probabilities are equally trustworthy within each demographic subgroup, addressing hidden reliability disparities that overall calibration metrics might miss.