Course contentsShow
← Machine Learning and Deep Learning
Lesson 536 of 3,53812. Model Validation and TuningFree lesson

Calibration in Practice

Implementing calibration: using sklearn's CalibratedClassifierCV, choosing validation sets, and evaluating improvements.

Calibration in Practice

You'll learn how to apply calibration techniques to real models using scikit-learn's built-in tools and evaluate whether calibration actually helps.

The Practical Workflow

Now that you understand why and how calibration methods work (Platt Scaling, Isotonic Regression, Temperature Scaling), it's time to actually implement them. Think of this as moving from theory to the workshop—you've learned about the tools, now you're using them on real wood.

Using CalibratedClassifierCV

Scikit-learn makes calibration straightforward with CalibratedClassifierCV. This wrapper takes your trained classifier and applies calibration:

from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier

# Train your base model
rf = RandomForestClassifier()
rf.fit(X_train, y_train)

# Apply calibration (method='sigmoid' for Platt, 'isotonic' for Isotonic Regression)
calibrated_rf = CalibratedClassifierCV(rf, method='sigmoid', cv=5)
calibrated_rf.fit(X_train, y_train)

Choosing Your Validation Set

The cv parameter determines how calibration learns the correction. Use separate held-out data (not your test set!) or cross-validation. If data is scarce, cv=5 splits your training data into folds for calibration.

Evaluating Improvements

Compare before and after using Brier Score and ECE (Expected Calibration Error):

from sklearn.metrics import brier_score_loss

# Before calibration
probs_uncal = rf.predict_proba(X_val)[:, 1]
brier_uncal = brier_score_loss(y_val, probs_uncal)

# After calibration
probs_cal = calibrated_rf.predict_proba(X_val)[:, 1]
brier_cal = brier_score_loss(y_val, probs_cal)

print(f"Brier before: {brier_uncal:.4f}, after: {brier_cal:.4f}")

Also plot reliability diagrams to visually confirm probabilities align with true frequencies.

Key Takeaway: Use CalibratedClassifierCV with held-out validation data, choose between Platt Scaling (sigmoid) or Isotonic Regression (isotonic), and verify improvements with Brier Score, ECE, and reliability diagrams.