Course contentsShow
← Data Science
Lesson 1207 of 2,14530. Exploratory Data Analysis (EDA) ProcessFree lesson

Missing Data Assessment and Strategy

Identify patterns in missing values, assess missingness mechanisms, and plan handling strategies.

Missing Data Assessment and Strategy

You'll learn how to spot patterns in missing data, diagnose why it's missing, and decide how to handle it before diving into analysis.

Why Missing Data Matters

Missing data isn't just annoying—it can bias your conclusions if you ignore it. Imagine a survey where high earners refuse to share their income: analyzing only complete responses would underestimate average income. Before filling gaps or dropping rows, you need to understand why data is missing.

Patterns in Missingness

Use frequency tables and visualizations to spot patterns:

  • Which columns have missing values?
  • How much is missing per column? (5% vs 70% matters)
  • Are certain rows missing data together? (e.g., entire time periods blank)
  • Do missingness patterns correlate with other variables? (e.g., older records missing newer fields)

You've already seen how to identify missing values with univariate analysis—now extend that to relationships between what's missing.

Missingness Mechanisms

Understanding why data is missing guides your strategy:

  • MCAR (Missing Completely at Random): No pattern—like random sensor failures. Safe to drop or impute.
  • MAR (Missing at Random): Missingness relates to observed data (e.g., young respondents skip "retirement age"). Imputation can work if you model the relationship.
  • MNAR (Missing Not at Random): Missingness relates to the unobserved value itself (e.g., people with debt hide income). Dangerous—simple fixes introduce bias.

Planning Your Strategy

Once you've assessed patterns and mechanisms, document your plan:

  • Drop: If <5% missing and MCAR
  • Impute: Replace with mean, median, mode, or modeled values
  • Flag: Create indicator columns for "was missing"
  • Keep separate: Analyze complete vs incomplete groups

Key Takeaway: Always explore why data is missing before deciding how to handle it—your strategy depends on the missingness mechanism, not just the amount.