Column Selection and Usecols
What You'll Learn
You'll learn how to load only the columns you actually need from large CSV files using the usecols parameter, dramatically reducing memory usage and speeding up data loading.
Why Load Only What You Need
When working with big datasets, reading the entire file into memory can be wasteful—especially when you only need a handful of columns for your analysis. Think of it like ordering a pizza: if you only want pepperoni and mushrooms, why pay for (and carry home) all 15 toppings?
The usecols parameter in pd.read_csv() lets you specify exactly which columns to load, ignoring the rest entirely. This happens during the read operation, so unwanted columns never enter memory at all.
How to Use Usecols
You can specify columns in three ways:
By column name (list of strings):
df = pd.read_csv('large_file.csv', usecols=['customer_id', 'purchase_amount', 'date'])
By column index (list of integers):
df = pd.read_csv('large_file.csv', usecols=[0, 3, 7]) # Load 1st, 4th, and 8th columns
By callable function:
df = pd.read_csv('large_file.csv', usecols=lambda col: col.startswith('sales_'))
Combining with Chunked Reading
You can combine usecols with chunked reading for even greater efficiency:
chunk_iterator = pd.read_csv('huge_file.csv',
usecols=['id', 'value'],
chunksize=50000)
This reads only two columns in manageable chunks—a powerful combination when memory is tight.
Key Takeaway: Use the usecols parameter in read_csv() to load only necessary columns, reducing memory footprint and improving I/O performance before data even enters memory.