Tokenizer Normalization and Preprocessing
You'll learn how cleaning and standardizing text before tokenization affects what your model sees and how it performs.
What Is Normalization and Preprocessing?
Before text gets split into tokens, it typically goes through a normalization pipeline that standardizes the input. Think of it like washing and sorting vegetables before cooking—you want consistent ingredients for consistent results.
Common preprocessing steps include:
Unicode Normalization
Different ways to represent the same character (like "é" as one character vs "e" + accent) get converted to a single standard form. Without this, your model might treat visually identical words as different tokens.
Lowercasing
Converting "Hello", "HELLO", and "hello" to the same form reduces vocabulary size and helps the model recognize they're the same word—but you lose information about proper nouns and sentence starts.
Accent Stripping
Removing accents ("café" → "cafe") simplifies the vocabulary but can change meaning in some languages or lose important distinctions.
Whitespace Handling
Normalizing tabs, newlines, and multiple spaces to single spaces creates cleaner, more consistent input.
Impact on Model Behavior
These choices directly affect your model:
- Aggressive normalization = smaller vocabulary, faster training, but potential information loss
- Minimal normalization = preserves nuance but creates more tokens and may struggle with variations
- Language matters: English tolerates lowercasing better than German (where nouns are capitalized)
The tokenizer you load already has these decisions baked in—BERT's WordPiece lowercases by default, while GPT-2's byte-level BPE preserves casing.
Key Takeaway: Normalization decisions made before tokenization determine what distinctions your model can learn—balance vocabulary efficiency against preserving meaningful information for your specific task.