Course contentsShow
← Machine Learning and Deep Learning
Lesson 1269 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Tokenizer Normalization and Preprocessing

Unicode normalization, lowercasing, accent stripping, and whitespace handling; impact on model behavior.

Tokenizer Normalization and Preprocessing

You'll learn how cleaning and standardizing text before tokenization affects what your model sees and how it performs.

What Is Normalization and Preprocessing?

Before text gets split into tokens, it typically goes through a normalization pipeline that standardizes the input. Think of it like washing and sorting vegetables before cooking—you want consistent ingredients for consistent results.

Common preprocessing steps include:

Unicode Normalization

Different ways to represent the same character (like "é" as one character vs "e" + accent) get converted to a single standard form. Without this, your model might treat visually identical words as different tokens.

Lowercasing

Converting "Hello", "HELLO", and "hello" to the same form reduces vocabulary size and helps the model recognize they're the same word—but you lose information about proper nouns and sentence starts.

Accent Stripping

Removing accents ("café" → "cafe") simplifies the vocabulary but can change meaning in some languages or lose important distinctions.

Whitespace Handling

Normalizing tabs, newlines, and multiple spaces to single spaces creates cleaner, more consistent input.

Impact on Model Behavior

These choices directly affect your model:

  • Aggressive normalization = smaller vocabulary, faster training, but potential information loss
  • Minimal normalization = preserves nuance but creates more tokens and may struggle with variations
  • Language matters: English tolerates lowercasing better than German (where nouns are capitalized)

The tokenizer you load already has these decisions baked in—BERT's WordPiece lowercases by default, while GPT-2's byte-level BPE preserves casing.

Key Takeaway: Normalization decisions made before tokenization determine what distinctions your model can learn—balance vocabulary efficiency against preserving meaningful information for your specific task.