Handling Missing Values
What you'll learn: How to detect, remove, and fill in missing data using pandas methods so your ML models don't break.
Why Missing Values Matter
Real-world datasets are messy. Surveys get skipped questions, sensors fail, data entry has gaps. In pandas, missing values appear as NaN (Not a Number). Most ML algorithms can't handle NaN directly—they'll throw errors or produce nonsense. Think of missing values like potholes on a road: you must either fill them or go around them.
Three Core Strategies
1. Detect Missing Values
Use .isna() or .isnull() to find where data is missing. These return Boolean masks showing True for missing spots.
2. Drop Missing Values
Use .dropna() to remove rows or columns with missing data. This is like throwing out damaged produce—quick but wasteful if you lose too much data.
# Drop any row with at least one NaN
df_clean = df.dropna()
# Drop columns with any NaN
df_clean = df.dropna(axis=1)
3. Fill Missing Values
Use .fillna() to replace NaN with a specific value (like the mean or median), or .interpolate() to estimate values based on surrounding data points.
# Fill with a constant
df['age'] = df['age'].fillna(0)
# Fill with column mean
df['salary'] = df['salary'].fillna(df['salary'].mean())
# Interpolate based on trend
df['temperature'] = df['temperature'].interpolate()
Choosing Your Strategy
Drop data when you have plenty to spare. Fill with statistics (mean, median) when values should be "typical." Interpolate for time-series or ordered data where nearby values predict missing ones.
Key Takeaway: Missing values must be addressed before training ML models—detect them with isna(), remove them with dropna(), or replace them with fillna() or interpolate() based on your data's nature.