Individual Fairness Metrics
What you'll learn: How to measure whether your AI system treats similar individuals similarly, regardless of their demographic group membership.
The Core Idea
While group fairness metrics (like demographic parity or equalized odds) focus on comparing outcomes across protected groups, individual fairness takes a different approach: similar people should receive similar predictions.
Think of it like a coffee shop loyalty program. Group fairness asks "Do men and women get the same average discount?" Individual fairness asks "Do two customers with nearly identical purchase histories get similar discounts, regardless of gender?"
How It Works
Individual fairness metrics evaluate whether your model's outputs are Lipschitz continuous with respect to a similarity metric on individuals. In plain terms:
- Define a similarity function that measures how alike two individuals are (based on relevant features)
- Measure the difference in model predictions for those individuals
- Check that similar inputs → similar outputs
For example, if Person A and Person B differ only slightly in credit history, income, and employment, their loan approval scores should also be close—even if they belong to different demographic groups.
Practical Considerations
The challenge is defining "similar." You need a domain-appropriate distance metric that:
- Captures meaningful similarity (not just numerical closeness)
- Excludes protected attributes when they shouldn't influence decisions
- Reflects real-world equivalence in your context
Unlike group metrics you've seen (demographic parity, equal opportunity), individual fairness is instance-level: it protects individuals from arbitrary treatment, not just groups from statistical disparities.
Key Takeaway: Individual fairness metrics ensure your AI treats similar people similarly by checking prediction consistency across comparable inputs, complementing group-level fairness measures with personalized protection.