Course contentsShow
← Python
Lesson 937 of 1,06015. Data Science FundamentalsFree lesson

Imputing Missing Data with Statistical Estimates

Apply mean, median, or mode calculations to fill gaps in numerical datasets while preserving the overall distribution of your variables.

In this lesson, you will learn how to replace missing values in your dataset with statistical estimates to maintain data integrity without discarding incomplete records.

When you encounter NaN (Not a Number) values in a DataFrame, dropping those rows can lead to a significant loss of information. Imputation allows you to "fill in the gaps" using the central tendency of the data. By using the mean, median, or mode, you retain the sample size while minimizing the impact of missingness on your overall statistical distribution.

Choosing the Right Estimator

  • Mean: Best for normally distributed numerical data without significant outliers.
  • Median: The robust choice when your data contains outliers that would otherwise skew the average.
  • Mode: The standard approach for categorical data or discrete numerical data where you need the most frequent value.

Implementation with Pandas

You can calculate these statistics using DataFrame methods and apply them to missing cells using the fillna() method. Suppose you have a DataFrame df representing employee salaries with some missing entries:

import pandas as pd
import numpy as np

# Calculate the median salary
median_salary = df['salary'].median()

# Fill missing values with the median
df['salary'] = df['salary'].fillna(median_salary)

By calculating the median and passing it to fillna(), you ensure that the gaps are replaced with a value that represents the center of your existing data. If you were working with a categorical column, such as "Department," you would instead calculate the mode()—which returns a Series—and select the first element to fill the missing spots: df['dept'].fillna(df['dept'].mode()[0]).

This technique prevents the distortion of your statistical summaries. While simple imputation does reduce the variance of your dataset (because you are inserting "average" values), it is a fundamental strategy for keeping your dataset intact during preprocessing.

Key Takeaway: Imputation allows you to preserve the utility of your dataset by replacing missing values with representative statistical estimates like the mean, median, or mode.