Course contentsShow
← Data Science
Lesson 1798 of 2,14542. Working with Big DataFree lesson

Memory Usage Profiling in Pandas

Learn to use memory_usage() and info() to identify memory bottlenecks and understand DataFrame memory consumption.

Memory Usage Profiling in Pandas

You'll learn how to identify where your DataFrame is consuming the most memory—essential for working with large datasets before they crash your system.

Why Memory Profiling Matters

When working with big data in Pandas, memory can be your biggest constraint. A DataFrame that looks innocent might secretly consume gigabytes of RAM. Before you reach for distributed tools like Spark or Dask, you need to understand where your memory is going. Think of it like checking your bank statement before deciding you need a second job—you might just need better spending habits.

The Core Tools: memory_usage() and info()

Pandas provides two essential methods for memory profiling:

memory_usage(deep=True) returns memory consumption per column in bytes. The deep=True parameter is crucial—it introspects object dtypes (like strings) to give you the actual memory footprint, not just the pointer size.

import pandas as pd

df = pd.DataFrame({
    'id': range(1000000),
    'category': ['A', 'B', 'C'] * 333334,
    'value': [1.5] * 1000000
})

# Get memory per column
memory_per_col = df.memory_usage(deep=True)
print(memory_per_col)
# Shows bytes consumed by each column

# Total memory in MB
total_mb = memory_per_col.sum() / 1024**2
print(f"Total: {total_mb:.2f} MB")

info(memory_usage='deep') provides a summary view combining dtype information with memory consumption—perfect for quick diagnosis.

df.info(memory_usage='deep')

Finding the Bottlenecks

Object columns (strings) often dominate memory usage. Compare columns to identify optimization targets—converting strings to categories or integers to smaller dtypes can yield massive savings.

Key Takeaway: Use memory_usage(deep=True) and info(memory_usage='deep') to diagnose DataFrame memory consumption column-by-column, identifying optimization opportunities before scaling to distributed computing.