Computing Fairness Metrics with Fairlearn and AIF360
You'll learn: How to use Python libraries to compute the fairness metrics you've studied in real ML projects.
Why Libraries Matter
Now that you understand metrics like demographic parity, equalized odds, and predictive parity, you need practical tools to compute them. Two leading Python libraries—Fairlearn and AIF360 (AI Fairness 360)—provide standardized implementations so you don't need to code metrics from scratch every time.
Fairlearn
Fairlearn (from Microsoft) focuses on scikit-learn compatibility and simplicity. It provides:
MetricFrame: Groups standard metrics (accuracy, precision) by sensitive features- Pre-built fairness metrics like
demographic_parity_difference()andequalized_odds_difference() - Mitigation algorithms (covered later)
Example workflow:
from fairlearn.metrics import MetricFrame, selection_rate
import pandas as pd
# Group metrics by sensitive attribute
mf = MetricFrame(
metrics=selection_rate,
y_true=y_test,
y_pred=predictions,
sensitive_features=sensitive_attr
)
print(mf.by_group) # Selection rate per group
print(mf.difference()) # Demographic parity difference
AIF360
IBM's AIF360 offers broader coverage of fairness concepts, including individual fairness measures and more exotic metrics. It uses its own dataset format and provides:
- Metrics for both binary and continuous predictions
- Counterfactual fairness checks
- Advanced bias detection tools
Both libraries compute the same core metrics but with different APIs and ecosystems.
Key Takeaway: Fairlearn and AIF360 turn theoretical fairness metrics into actionable measurements—use Fairlearn for quick scikit-learn integration, AIF360 for comprehensive bias audits across multiple definitions of fairness.