Course contentsShow
← Python
Lesson 935 of 1,06015. Data Science FundamentalsFree lesson

Identifying Missing Values in Datasets

Learn to utilize Pandas methods to detect null entries and visualize the distribution of missing data across your entire dataframe.

In this lesson, you will learn how to systematically identify missing values within a Pandas DataFrame to maintain data integrity before analysis.

Detecting Null Entries

When working with real-world datasets, data is rarely perfect. Pandas uses the isnull() (or its alias isna()) method to create a boolean mask of the same shape as your DataFrame. Every cell containing a null value (like NaN or None) is marked as True, while valid data is marked as False.

Because a large DataFrame can have thousands of entries, viewing the entire boolean matrix is inefficient. Instead, we typically aggregate these values to get a quick summary:

import pandas as pd
import numpy as np

# Creating a sample DataFrame with missing values
df = pd.DataFrame({
    'A': [1, 2, np.nan],
    'B': [4, np.nan, np.nan],
    'C': [7, 8, 9]
})

# Summing the True values (treated as 1) along each column
missing_counts = df.isnull().sum()
print(missing_counts)

By calling .sum() on the boolean mask, Pandas aggregates the missing values down each column, instantly showing you exactly which features have the most gaps.

Visualizing Distribution

To get a more granular view of where missing data clusters—for example, if certain rows are missing entire sets of information—you can use boolean masking combined with visual techniques.

A simple but effective way to visualize the distribution is to compare the shape of your filtered data against the original. If you want to identify rows where data is missing, you can use .any() across the columns (axis=1):

# Identify rows that have at least one missing value
rows_with_nulls = df[df.isnull().any(axis=1)]

By filtering the DataFrame this way, you isolate the specific rows that require attention, allowing you to decide whether to impute these values or drop the affected rows entirely. Combining these methods ensures that you don't just know that you have missing data, but exactly where and how much exists across your entire dataset.

Key Takeaway: Use df.isnull().sum() to quantify missing data per column and df.isnull().any(axis=1) to isolate rows that require cleaning.