Brier Score for Probabilistic Calibration
What you'll learn: How to measure how well your model's predicted probabilities match actual outcomes using a single number.
What Is the Brier Score?
The Brier Score measures the mean squared difference between predicted probabilities and actual outcomes. Think of it as asking: "When my model says there's a 70% chance of rain, does it actually rain 70% of those times?"
Unlike metrics that only care if you got the label right (yes/no), Brier Score evaluates the quality of your probability estimates. A well-calibrated model doesn't just predict correctly—it expresses the right level of confidence.
How It Works
For each prediction, you calculate the squared difference between:
- The predicted probability (e.g., 0.7 for "likely positive")
- The actual outcome (1 if it happened, 0 if it didn't)
Then you average these squared differences across all predictions:
Brier Score = (1/N) × Σ(predicted_probability - actual_outcome)²
Understanding the Score
- Range: 0 to 1 (lower is better)
- Perfect score: 0.0 (predicted probabilities exactly match reality)
- Worst score: 1.0 (maximally wrong probabilities)
- Random baseline: ~0.25 for balanced binary problems
Example Intuition
If your model predicts 90% probability for positive class, but the actual outcome is 0 (negative), you get a large penalty: (0.9 - 0)² = 0.81. If it predicts 0.1 and actual is 0, the penalty is small: (0.1 - 0)² = 0.01.
Key Takeaway: The Brier Score quantifies probabilistic calibration by penalizing both incorrect predictions and overconfident/underconfident probability estimates—essential for applications where decision-making relies on probability values, not just class labels.