Course contentsShow
← Python
Lesson 941 of 1,06015. Data Science FundamentalsFree lesson

Splitting and Extracting Substrings from Columns

Learn to parse complex string columns into multiple structured features using delimiter-based splitting and regular expression extraction.

In this lesson, you will learn how to parse unstructured, complex text data stored within DataFrame columns into clean, multi-column features.

When working with real-world datasets, you often encounter columns that pack multiple pieces of information into a single string, such as a "Full Name" or a "Product Code" that contains a category, ID, and region. To analyze this data, you need to break these strings apart.

Splitting by Delimiters

The simplest way to parse strings is using the str.split() accessor. By identifying a common character—a delimiter—you can tell pandas to divide the string into a list of components. If you set expand=True, pandas automatically creates a new DataFrame with each piece of the string in its own column.

import pandas as pd

df = pd.DataFrame({'Contact': ['John_Doe', 'Jane_Smith']})

# Split the string at the underscore and expand into two separate columns
df[['First', 'Last']] = df['Contact'].str.split('_', expand=True)

Extracting with Regular Expressions

Sometimes data is not neatly separated by a simple character. For more complex patterns, you use str.extract(), which utilizes regular expressions (regex). This allows you to define a "template" using capture groups (parentheses). When pandas finds a match, it pulls out exactly what is inside the parentheses into dedicated columns.

For example, if you have a serial number like "ID:99-AX", you can extract the numeric ID and the character code simultaneously:

df = pd.DataFrame({'Serial': ['ID:01-AA', 'ID:02-BB']})

# Extract the digits and the characters into two named columns
pattern = r'ID:(\d+)-([A-Z]+)'
df[['Number', 'Code']] = df['Serial'].str.extract(pattern)

By leveraging expand=True for splitting and regex capture groups for extraction, you transform raw, cluttered text into structured data ready for numerical analysis. This prevents "data bloat" where a single column holds too much semantic information, making your features much easier to filter and aggregate.

Key Takeaway: Use str.split(expand=True) for simple delimiter-based parsing and str.extract() with regex groups to decompose complex, non-uniform string patterns into structured, multi-column features.