Normalizing Input Text
What you'll learn: How text normalization decisions in preprocessing directly influence tokenizer consistency, vocabulary efficiency, and ultimately model behavior.
Why Normalize Text?
Before tokenization, raw text arrives in many forms: mixed case, inconsistent whitespace, varied Unicode representations. Normalization is the process of converting text into a canonical form so your tokenizer treats semantically identical inputs consistently.
Think of it like standardizing addresses before mailing letters—"123 Main St." and "123 MAIN STREET" should be recognized as the same destination.
Key Normalization Decisions
Unicode Normalization
Unicode characters can be represented multiple ways. The character "é" might be:
- A single precomposed character (U+00E9)
- Base "e" + combining accent (U+0065 + U+0301)
NFC (Composed) and NFD (Decomposed) are two standard forms. Picking one ensures "café" is always tokenized identically, regardless of how it was typed.
Case Handling
Should your model treat "Apple" (fruit) and "Apple" (company) the same? Lowercasing simplifies the vocabulary but loses information. Many modern LLMs keep original case to preserve proper nouns and sentence boundaries, accepting a larger vocabulary in exchange.
Whitespace Treatment
Collapsing multiple spaces, removing trailing whitespace, and deciding whether tabs equal spaces prevents tokenization drift. Inconsistent whitespace creates distinct tokens for functionally identical text.
Impact on Tokenizer Behavior
These choices compound: unnormalized text leads to vocabulary bloat (multiple tokens for the same concept), inconsistent training signal, and unpredictable inference behavior. Your normalization strategy must match between training data preparation and deployment.
Key Takeaway: Text normalization (Unicode form, case policy, whitespace rules) must be applied consistently during preprocessing to ensure tokenizer stability, vocabulary efficiency, and predictable model behavior across training and inference.