Data Type Conversion and Categorical Data
What you'll learn: How to convert data types and use categorical types to make your datasets leaner and ready for machine learning.
Why Data Types Matter
When you load data into a Pandas DataFrame, columns automatically get assigned data types (dtypes). Sometimes these defaults waste memory or aren't suitable for ML. For example, a column with values like "Small", "Medium", "Large" might be stored as regular text (object dtype), consuming far more memory than necessary.
Converting Data Types
You can explicitly convert column types using astype():
# Convert numeric column stored as string
df['age'] = df['age'].astype('int64')
# Convert to float
df['price'] = df['price'].astype('float32')
Using smaller numeric types (like float32 instead of float64) can significantly reduce memory usage for large datasets.
Categorical Data: The Memory Saver
When a column has a limited number of repeated values (like product categories, country names, or size labels), convert it to category dtype:
df['size'] = df['size'].astype('category')
Think of it like this: Instead of storing the word "Large" a million times, Pandas stores "Large" once and uses small numeric codes to reference it throughout the column—like a lookup table.
This optimization:
- Reduces memory dramatically (sometimes by 90%+)
- Speeds up operations on that column
- Prepares data for encoding (most ML algorithms need numbers, not text)
Later in your ML pipeline, you'll encode these categories into numbers that algorithms can process.
Key Takeaway: Use astype() to convert dtypes for efficiency and correctness; convert repeated string columns to category dtype to save memory and prepare for numerical encoding required by ML models.