Text Files: Fixed-Width and Custom Parsers
What you'll learn: How to read structured text files where columns don't use delimiters, and build custom parsers for unusual file formats.
Not all text data comes nicely separated by commas or tabs. Sometimes you'll encounter fixed-width format (FWF) files, where each column occupies a specific number of characters, padded with spaces. Think of it like an old-school typewritten table: the first 10 characters are always the name, the next 5 are the age, and so on—no delimiters needed.
Reading Fixed-Width Files with read_fwf()
Pandas provides read_fwf() specifically for these files. You can specify column widths explicitly or let Pandas infer them:
import pandas as pd
# Specify exact column widths
df = pd.read_fwf('data.txt', widths=[10, 5, 8])
# Or specify column positions (start, end)
df = pd.read_fwf('data.txt', colspecs=[(0, 10), (10, 15), (15, 23)])
# Let Pandas infer widths (works for well-aligned files)
df = pd.read_fwf('data.txt')
Custom Parsers for Irregular Formats
Sometimes files don't follow any standard format—mixed delimiters, comment lines, headers scattered throughout. Here you need a custom parser: read the file line-by-line, apply your own logic, and build the DataFrame manually:
data = []
with open('weird_file.txt', 'r') as f:
for line in f:
if line.startswith('#'): # Skip comments
continue
# Custom parsing logic
parts = line.strip().split('|')
data.append(parts)
df = pd.DataFrame(data, columns=['col1', 'col2', 'col3'])
This approach gives you complete control when standard readers fail.
Key Takeaway: Use read_fwf() with widths or colspecs for fixed-width files, and write custom line-by-line parsers when dealing with truly irregular text formats that don't fit standard patterns.