Course contentsShow
← Machine Learning and Deep Learning
Lesson 3301 of 3,53872. Bias and FairnessFree lesson

Measuring Bias in Rankings and Recommendations

Fairness metrics for ordered outputs: exposure, representation, and position bias.

Measuring Bias in Rankings and Recommendations

What you'll learn: How to quantify fairness in systems that produce ordered lists, like search results or product recommendations, using exposure, representation, and position bias metrics.

Why Rankings Need Different Fairness Metrics

Traditional fairness metrics assume binary or categorical predictions, but many ML systems produce ordered lists—search results, job candidate rankings, movie recommendations. Here, position matters dramatically: items at the top get far more attention than those buried on page 3, even if all items are technically "recommended."

Core Fairness Metrics for Ordered Outputs

Exposure

Exposure measures how much visibility each item or group receives based on position. Items ranked higher receive exponentially more attention (users rarely scroll down). Fair exposure means protected groups should receive visibility proportional to their relevance or representation.

Analogy: Imagine a job board that ranks candidates. If qualified women consistently appear on page 2 while equally qualified men dominate page 1, women receive less "exposure" to employers—even if both groups are technically "recommended."

Representation

Representation examines whether different groups appear in the top-k results proportionally. A recommendation system might include diverse groups but still fail if minorities never appear in the critical top-10 positions.

Position Bias

Position bias acknowledges that users trust and click top-ranked items more, regardless of actual quality. Fairness requires correcting for this: if a system learns from biased click data (where position influenced clicks), it perpetuates the bias by continuing to rank historically-favored groups higher.

Key Takeaway

Rankings require specialized fairness metrics because position creates visibility disparities—fair rankings must consider not just who is included, but where they appear and how much exposure each group receives across the ordered list.