Course contentsShow
← Machine Learning and Deep Learning
Lesson 169 of 3,5382. Machine Learning Mindset and Core ConceptsFree lesson

Handling Missing Values

Detect, drop, and fill NaN values using fillna, dropna, and interpolate methods.

Handling Missing Values

What you'll learn: How to detect, remove, and fill in missing data using pandas methods so your ML models don't break.

Why Missing Values Matter

Real-world datasets are messy. Surveys get skipped questions, sensors fail, data entry has gaps. In pandas, missing values appear as NaN (Not a Number). Most ML algorithms can't handle NaN directly—they'll throw errors or produce nonsense. Think of missing values like potholes on a road: you must either fill them or go around them.

Three Core Strategies

1. Detect Missing Values

Use .isna() or .isnull() to find where data is missing. These return Boolean masks showing True for missing spots.

2. Drop Missing Values

Use .dropna() to remove rows or columns with missing data. This is like throwing out damaged produce—quick but wasteful if you lose too much data.

# Drop any row with at least one NaN
df_clean = df.dropna()

# Drop columns with any NaN
df_clean = df.dropna(axis=1)

3. Fill Missing Values

Use .fillna() to replace NaN with a specific value (like the mean or median), or .interpolate() to estimate values based on surrounding data points.

# Fill with a constant
df['age'] = df['age'].fillna(0)

# Fill with column mean
df['salary'] = df['salary'].fillna(df['salary'].mean())

# Interpolate based on trend
df['temperature'] = df['temperature'].interpolate()

Choosing Your Strategy

Drop data when you have plenty to spare. Fill with statistics (mean, median) when values should be "typical." Interpolate for time-series or ordered data where nearby values predict missing ones.

Key Takeaway: Missing values must be addressed before training ML models—detect them with isna(), remove them with dropna(), or replace them with fillna() or interpolate() based on your data's nature.