Course contentsShow
← Python
Lesson 926 of 1,06015. Data Science FundamentalsFree lesson

Filtering DataFrame Rows with Boolean Conditions

Apply conditional expressions to filter rows and extract subsets of data that meet specific logical criteria.

In this lesson, you will learn how to extract specific subsets of data from a pandas DataFrame by applying logical filters directly to your rows.

The Power of Boolean Indexing

When working with DataFrames, you often need to isolate rows that satisfy a specific business logic, such as finding all customers with a balance over $1,000 or identifying entries from a particular region. While we previously explored Boolean masking with NumPy arrays, DataFrames allow us to apply this same logic using column names to create readable, powerful queries.

To filter rows, you create a "Boolean mask"—a Series of True and False values generated by a conditional expression. When you pass this mask into the DataFrame's indexing operator [], pandas returns only the rows where the mask evaluates to True.

Practical Implementation

Consider a DataFrame df containing employee data. If you want to filter for employees in the 'Engineering' department with a salary greater than $80,000, you can combine conditions using the bitwise & (AND) operator. Note that each individual condition must be wrapped in parentheses:

import pandas as pd

# Assume 'df' is our DataFrame
# Create a mask for rows meeting both criteria
mask = (df['department'] == 'Engineering') & (df['salary'] > 80000)

# Filter the DataFrame using the mask
high_earning_engineers = df[mask]

Why This Matters

This approach is more efficient and readable than iterating through rows with a loop. Because pandas leverages the vectorized operations we covered earlier, filtering large datasets becomes nearly instantaneous. By applying these boolean conditions, you transform a large, monolithic dataset into a refined subset ready for further analysis, aggregation, or visualization. Always ensure that your conditions are logically sound and that you use the bitwise operators (&, |, ~) rather than Python’s standard and/or keywords, which are not designed to handle array-like structures.

Key Takeaway: Boolean indexing allows you to query DataFrames by passing a mask of True/False values to the indexing operator, enabling efficient, vectorized subsetting of your data based on complex logical criteria.