Course contentsShow
← Data Science
Lesson 1880 of 2,14544. Data Ethics and PrivacyFree lesson

Measurement and Label Bias

Recognizing bias from imperfect measurement instruments, subjective labeling, and inconsistent data collection standards.

Measurement and Label Bias

What you'll learn: How imperfect measurement tools, subjective labeling decisions, and inconsistent data collection practices introduce bias into your datasets.

The Problem with Imperfect Instruments

Even when you've sampled data correctly (avoiding selection bias), the way you measure or label observations can systematically distort your data. Measurement bias occurs when your data collection instruments, procedures, or definitions consistently produce inaccurate values. Label bias happens when subjective human judgment creates inconsistent or skewed labels for supervised learning.

Common Sources

Imperfect measurement instruments: A broken thermometer that reads 2°C high introduces systematic error. In data science, this might be a biased survey question, a miscalibrated sensor, or a flawed proxy metric (using clicks as a measure of satisfaction when they don't align).

Subjective labeling: When humans label training data—tagging images, rating sentiment, or classifying documents—their personal biases, cultural backgrounds, and varying interpretations create inconsistency. Different annotators might label the same borderline case differently.

Inconsistent standards: If your data collection team changes definitions midway (e.g., "What counts as 'active user'?"), or if different sources use different measurement protocols, you're comparing apples to oranges.

Real-World Impact

Imagine training a hiring model on performance ratings. If some managers consistently rate everyone lower while others inflate scores, your labels reflect manager bias more than actual performance. Or consider medical diagnosis data where one hospital uses stricter criteria than another—your model learns the hospitals' standards, not the underlying conditions.

Key Takeaway: Measurement and label bias arise from how you collect and annotate data, not just what data you collect. Always audit your instruments, standardize labeling processes with clear guidelines, and document any changes in measurement protocols to identify potential systematic errors.