Statistical Significance in Slice Evaluation
What you'll learn: How to determine whether observed performance differences across data slices are real signals or just random noise, while avoiding false discoveries when testing many slices.
The Multiple Testing Challenge
When you evaluate your model across dozens or hundreds of slices (demographic groups, feature ranges, temporal periods), you're running many statistical comparisons simultaneously. If you use standard significance thresholds (like p < 0.05) for each slice independently, you'll find "significant" differences purely by chance.
Analogy: If you flip a coin 100 times looking for heads streaks, you'll inevitably find some runs of 5+ heads just by randomness. Similarly, testing 100 slices will surface ~5 "significant" findings at the 0.05 level even when nothing meaningful differs.
Addressing the Problem
To determine if slice performance differences are meaningful, apply multiple testing corrections:
- Bonferroni correction: Divide your significance threshold (e.g., 0.05) by the number of slices tested. Conservative but simple.
- Benjamini-Hochberg (FDR control): Controls the false discovery rate, less conservative than Bonferroni.
- Bootstrap confidence intervals: Resample your evaluation data to establish empirical confidence bounds for each slice's metric.
Beyond p-values
Statistical significance isn't enough. A difference can be statistically significant but practically irrelevant (large sample, tiny effect). Always consider:
- Effect size: Is the accuracy drop 0.1% or 10%? Magnitude matters.
- Sample size per slice: Small slices yield unstable estimates and wider confidence intervals.
- Business impact: Does this difference affect user experience or fairness?
Key Takeaway: When evaluating performance across many slices, apply multiple testing corrections to avoid false alarms, and always combine statistical significance with practical significance and domain context to identify truly problematic model behavior.