Course contentsShow
← Machine Learning and Deep Learning
Lesson 1650 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Normalizing Input Text

Unicode normalization, case handling, whitespace treatment, and preprocessing decisions that affect tokenizer behavior.

Normalizing Input Text

What you'll learn: How text normalization decisions in preprocessing directly influence tokenizer consistency, vocabulary efficiency, and ultimately model behavior.

Why Normalize Text?

Before tokenization, raw text arrives in many forms: mixed case, inconsistent whitespace, varied Unicode representations. Normalization is the process of converting text into a canonical form so your tokenizer treats semantically identical inputs consistently.

Think of it like standardizing addresses before mailing letters—"123 Main St." and "123 MAIN STREET" should be recognized as the same destination.

Key Normalization Decisions

Unicode Normalization

Unicode characters can be represented multiple ways. The character "é" might be:

  • A single precomposed character (U+00E9)
  • Base "e" + combining accent (U+0065 + U+0301)

NFC (Composed) and NFD (Decomposed) are two standard forms. Picking one ensures "café" is always tokenized identically, regardless of how it was typed.

Case Handling

Should your model treat "Apple" (fruit) and "Apple" (company) the same? Lowercasing simplifies the vocabulary but loses information. Many modern LLMs keep original case to preserve proper nouns and sentence boundaries, accepting a larger vocabulary in exchange.

Whitespace Treatment

Collapsing multiple spaces, removing trailing whitespace, and deciding whether tabs equal spaces prevents tokenization drift. Inconsistent whitespace creates distinct tokens for functionally identical text.

Impact on Tokenizer Behavior

These choices compound: unnormalized text leads to vocabulary bloat (multiple tokens for the same concept), inconsistent training signal, and unpredictable inference behavior. Your normalization strategy must match between training data preparation and deployment.

Key Takeaway: Text normalization (Unicode form, case policy, whitespace rules) must be applied consistently during preprocessing to ensure tokenizer stability, vocabulary efficiency, and predictable model behavior across training and inference.