In this lesson, you will learn how to drastically shrink your DataFrame’s memory footprint and boost computational speed by converting high-cardinality string columns into the category data type.
The Problem with Strings
By default, Pandas stores string data using the object dtype. In this format, every single string in a column is stored as an independent Python object in memory. If you have a column like "City" with one million rows, and that city is "New York," Pandas stores the characters for "New York" a million separate times, along with extra metadata for each object. This is highly inefficient.
The Categorical Solution
The category type works by treating your data as a set of unique values. Pandas creates a mapping (a dictionary of sorts) where it assigns each unique string a small integer code. Instead of storing "New York" one million times, it stores the integer 1 one million times, and keeps a single lookup table for the values.
Consider this example:
import pandas as pd
import numpy as np
# A large column with repetitive string data
df = pd.DataFrame({'city': ['New York'] * 1000000})
# Check original memory usage
print(df.memory_usage(deep=True))
# Convert to category
df['city'] = df['city'].astype('category')
# Check optimized memory usage
print(df.memory_usage(deep=True))
When to Use It
This technique is most effective when the number of unique strings (cardinality) is small relative to the total number of rows. If every single row contains a unique string (like a full name or a UUID), converting to category might actually increase memory usage, as the overhead of the lookup table outweighs the savings.
However, for columns like "State," "Department," "Status," or any repetitive categorical label, the memory savings are often 80-90% or more. Beyond just saving space, many Pandas operations—like sorting or grouping by these columns—become significantly faster because the underlying engine can perform arithmetic on the integer codes instead of performing expensive string comparisons.
Key Takeaway: Converting columns with low-to-moderate cardinality from object to category minimizes redundant storage and accelerates downstream performance by leveraging integer-based lookups.