In this lesson, you will learn how to parse unstructured, complex text data stored within DataFrame columns into clean, multi-column features.
When working with real-world datasets, you often encounter columns that pack multiple pieces of information into a single string, such as a "Full Name" or a "Product Code" that contains a category, ID, and region. To analyze this data, you need to break these strings apart.
Splitting by Delimiters
The simplest way to parse strings is using the str.split() accessor. By identifying a common character—a delimiter—you can tell pandas to divide the string into a list of components. If you set expand=True, pandas automatically creates a new DataFrame with each piece of the string in its own column.
import pandas as pd
df = pd.DataFrame({'Contact': ['John_Doe', 'Jane_Smith']})
# Split the string at the underscore and expand into two separate columns
df[['First', 'Last']] = df['Contact'].str.split('_', expand=True)
Extracting with Regular Expressions
Sometimes data is not neatly separated by a simple character. For more complex patterns, you use str.extract(), which utilizes regular expressions (regex). This allows you to define a "template" using capture groups (parentheses). When pandas finds a match, it pulls out exactly what is inside the parentheses into dedicated columns.
For example, if you have a serial number like "ID:99-AX", you can extract the numeric ID and the character code simultaneously:
df = pd.DataFrame({'Serial': ['ID:01-AA', 'ID:02-BB']})
# Extract the digits and the characters into two named columns
pattern = r'ID:(\d+)-([A-Z]+)'
df[['Number', 'Code']] = df['Serial'].str.extract(pattern)
By leveraging expand=True for splitting and regex capture groups for extraction, you transform raw, cluttered text into structured data ready for numerical analysis. This prevents "data bloat" where a single column holds too much semantic information, making your features much easier to filter and aggregate.
Key Takeaway: Use str.split(expand=True) for simple delimiter-based parsing and str.extract() with regex groups to decompose complex, non-uniform string patterns into structured, multi-column features.