Isotonic Regression for Calibration
What you'll learn: How to use isotonic regression as a flexible, non-parametric method to calibrate your model's predicted probabilities without assuming any specific functional form.
The Core Idea
While Platt Scaling fits a specific parametric function (logistic) to map your model's scores to calibrated probabilities, isotonic regression takes a more flexible approach. It learns a monotonic (non-decreasing) step function that maps uncalibrated scores to calibrated probabilities directly from data, making no assumptions about the relationship's shape.
Think of it like building a staircase: isotonic regression groups your predictions into bins and adjusts each bin's height to ensure that predicted probabilities match actual frequencies, with the constraint that going up the stairs (higher predicted scores) never leads downward (lower calibrated probabilities).
When to Use It
Isotonic regression shines when:
- Your calibration curve isn't sigmoid-shaped (Platt Scaling's assumption)
- You have sufficient calibration data (it's more data-hungry than Platt Scaling)
- Your model produces rankings that are good but probabilities that are systematically wrong in complex ways
The Trade-off
Because isotonic regression is non-parametric, it's more flexible but also more prone to overfitting on small calibration sets. It essentially memorizes the calibration data's shape, which is powerful when you have enough examples but risky otherwise.
The monotonicity constraint prevents complete chaos—it ensures that if model A predicts higher than model B, the calibrated probability for A won't be lower than B's—but you still need hundreds or thousands of calibration examples for stable results.
Key Takeaway: Isotonic regression provides flexible, non-parametric calibration by learning a monotonic mapping from scores to probabilities, ideal when calibration curves are non-sigmoid but requires more data than parametric methods like Platt Scaling.