Choosing Fairness Criteria for Your Problem
What you'll learn: You'll discover how to systematically select the right fairness definition for your specific ML application by weighing domain requirements, stakeholder needs, and potential harms.
The Selection Challenge
You've learned many fairness criteria—demographic parity, equal opportunity, equalized odds, predictive parity, calibration, counterfactual fairness—but which should you actually use? There's no universal answer. Choosing the right fairness metric is itself a critical design decision that requires careful analysis beyond pure mathematics.
A Framework for Selection
1. Identify Stakeholders and Harms
Who is affected by your model's decisions? What specific harms could occur? A lending model might harm qualified applicants who are denied (false negatives) differently than it harms the bank through bad loans (false positives).
2. Understand the Decision Context
Is your model making selective decisions (college admissions—limited slots) or assistive decisions (medical diagnosis—helping everyone)? Selective contexts often prioritize equal opportunity, while assistive contexts may focus on calibration or predictive parity.
3. Consider Legal and Regulatory Requirements
Some jurisdictions mandate specific fairness approaches. Employment law might require demonstrating equal opportunity across protected groups.
4. Account for Base Rates
When outcome prevalence differs across groups naturally (not due to bias), demographic parity may be inappropriate. Medical conditions with different base rates by age group need different criteria than hiring decisions.
5. Embrace Trade-offs
Remember the impossibility theorem: you often can't satisfy multiple fairness criteria simultaneously. Document which harms you're prioritizing and why.
Practical Decision Path
Think of it like choosing safety features for different vehicles: a school bus prioritizes passenger protection (equal opportunity for students), while an ambulance optimizes speed to help patients (assistive context requiring calibration). The "right" choice depends entirely on the mission.
Key Takeaway: Selecting fairness criteria requires analyzing your specific domain context, consulting stakeholders about which harms matter most, understanding legal constraints, and explicitly documenting the trade-offs you accept—it's a sociotechnical decision, not just a mathematical one.