Merging and Joining DataFrames
What you'll learn: How to combine multiple datasets from different sources into a single DataFrame for more powerful machine learning features.
Why Merging Matters in ML
In real-world ML projects, your data rarely arrives in one perfect table. You might have customer demographics in one file, purchase history in another, and product details in a third. To build effective models, you need to integrate these datasets—that's where merging and joining come in.
Think of it like assembling puzzle pieces: each dataset contains part of the picture, and you need to connect them using common identifiers (like customer IDs or product codes).
Three Core Pandas Operations
merge()
Combines DataFrames based on shared columns (like SQL joins). You specify which column(s) to match on:
# Combine customer info with their orders using customer_id
combined = customer_df.merge(orders_df, on='customer_id')
join()
Similar to merge, but typically uses the DataFrame index for matching:
# Join using index alignment
result = df1.join(df2)
concat()
Stacks DataFrames vertically (rows) or horizontally (columns) without needing matching keys:
# Stack datasets from different months
all_data = pd.concat([january_df, february_df, march_df])
ML Use Case Example
Imagine predicting customer churn. You merge customer demographics with transaction history to create rich features—combining age, location, purchase frequency, and average order value into one training dataset.
Key Takeaway: Merging, joining, and concatenating DataFrames let you integrate multiple data sources into unified feature sets, essential for building ML models with comprehensive information from different tables or files.