Selection Bias and Sampling Bias
What you'll learn: How flawed data collection methods and missing populations create non-representative samples that skew your analysis.
The Core Problem
Selection bias occurs when the data you collect doesn't accurately represent the population you're trying to study. This happens through two main pathways: sampling bias (who gets included in your dataset) and selection mechanisms (systematic exclusions during data collection).
Think of it like surveying "typical Americans" by only calling landline phones during business hours. You've systematically excluded mobile-only users, working people, and younger demographics—your sample is biased.
Common Sources
Sampling Bias arises when:
- Your data collection method favors certain subgroups (online surveys exclude those without internet)
- Voluntary response creates self-selection (angry customers review more than satisfied ones)
- Convenience sampling ignores hard-to-reach populations
Selection Bias happens when:
- Historical data reflects past discriminatory practices
- Dropout or attrition affects certain groups differently
- Data collection systems are only deployed in specific locations or contexts
Real-World Impact
A hiring algorithm trained on historical employee data inherits selection bias if past hiring was discriminatory. A health model trained only on patients who sought treatment misses asymptomatic populations. A credit model using zip codes as features encodes historical redlining patterns.
The danger: your model will be very accurate at predicting patterns in your biased sample, but fail catastrophically when applied to the real world.
Prevention Strategies
- Document your data collection methodology thoroughly
- Compare your sample demographics to known population statistics
- Identify missing or underrepresented groups
- Use stratified sampling or weighting techniques when possible
- Question whether excluded populations differ systematically
Key Takeaway: Non-representative samples arise from how data is collected, not just what's collected—always ask "who's missing from this dataset and why?"