Calibration in Practice
You'll learn how to apply calibration techniques to real models using scikit-learn's built-in tools and evaluate whether calibration actually helps.
The Practical Workflow
Now that you understand why and how calibration methods work (Platt Scaling, Isotonic Regression, Temperature Scaling), it's time to actually implement them. Think of this as moving from theory to the workshop—you've learned about the tools, now you're using them on real wood.
Using CalibratedClassifierCV
Scikit-learn makes calibration straightforward with CalibratedClassifierCV. This wrapper takes your trained classifier and applies calibration:
from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier
# Train your base model
rf = RandomForestClassifier()
rf.fit(X_train, y_train)
# Apply calibration (method='sigmoid' for Platt, 'isotonic' for Isotonic Regression)
calibrated_rf = CalibratedClassifierCV(rf, method='sigmoid', cv=5)
calibrated_rf.fit(X_train, y_train)
Choosing Your Validation Set
The cv parameter determines how calibration learns the correction. Use separate held-out data (not your test set!) or cross-validation. If data is scarce, cv=5 splits your training data into folds for calibration.
Evaluating Improvements
Compare before and after using Brier Score and ECE (Expected Calibration Error):
from sklearn.metrics import brier_score_loss
# Before calibration
probs_uncal = rf.predict_proba(X_val)[:, 1]
brier_uncal = brier_score_loss(y_val, probs_uncal)
# After calibration
probs_cal = calibrated_rf.predict_proba(X_val)[:, 1]
brier_cal = brier_score_loss(y_val, probs_cal)
print(f"Brier before: {brier_uncal:.4f}, after: {brier_cal:.4f}")
Also plot reliability diagrams to visually confirm probabilities align with true frequencies.
Key Takeaway: Use CalibratedClassifierCV with held-out validation data, choose between Platt Scaling (sigmoid) or Isotonic Regression (isotonic), and verify improvements with Brier Score, ECE, and reliability diagrams.