Course contentsShow
← Machine Learning and Deep Learning
Lesson 1244 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Preprocessing Before Tokenization

Text normalization steps: lowercasing, Unicode handling, whitespace, and punctuation considerations.

Preprocessing Before Tokenization

What you'll learn: How to clean and normalize raw text before splitting it into tokens, ensuring consistent and meaningful input for your model.

Why Preprocessing Matters

Before you break text into tokens, you need to clean it up. Think of it like washing vegetables before chopping them—you want to remove dirt and inconsistencies first. Raw text from the internet, books, or user input contains irregularities that can confuse your tokenizer or create unnecessary vocabulary entries.

Core Text Normalization Steps

Lowercasing

Converting all text to lowercase treats "Hello," "hello," and "HELLO" as the same word. This reduces vocabulary size and helps the model recognize that these are semantically identical. However, for some tasks (like named entity recognition), case matters—"Apple" (company) vs "apple" (fruit).

Unicode Handling

Text may contain different Unicode representations of the same character (e.g., é can be stored as one character or as e + accent). Unicode normalization standardizes these variations so your model sees them consistently.

Whitespace Normalization

Extra spaces, tabs, and newlines should typically be collapsed into single spaces. Multiple spaces between words create inconsistent input and waste token budget.

Punctuation Considerations

Decide whether to keep, remove, or separate punctuation. "Hello!" might become "Hello !" or "Hello" depending on your task. For sentiment analysis, "!!!" might carry meaning. For topic modeling, it might just add noise.

The Right Order

Clean your text before tokenization. If you lowercase after tokenization, you've already created separate vocabulary entries for "Hello" and "hello"—defeating the purpose.

Key Takeaway: Preprocessing transforms messy real-world text into clean, normalized input, reducing vocabulary size and improving model consistency before any tokenization occurs.