Why Subword Tokenization?
You'll learn: Why splitting text into subword units solves critical problems that character-level and word-level tokenization cannot handle alone.
The Problem with Extremes
You've already seen two tokenization approaches, but each has deal-breaking flaws:
Word-level tokenization treats every unique word as a separate token. This creates massive vocabularies and explodes when encountering new words (the OOV problem). Words like "running," "runner," and "runs" get entirely separate entries with no shared understanding.
Character-level tokenization eliminates OOV issues—every word is just a sequence of known characters. But now your model must learn to build words from scratch every time, making sequences much longer and relationships harder to capture. The word "transformer" becomes 11 separate tokens instead of one meaningful unit.
The Subword Solution
Subword tokenization finds the middle ground. It breaks text into meaningful chunks smaller than words but larger than characters. Think of it like recognizing word parts: "unhappiness" might become ["un", "happi", "ness"].
This approach gives you the best of both worlds:
- Rare words? Break them into known subwords instead of marking as
<UNK> - Common words? Keep them as single tokens for efficiency
- Related words? Share subword pieces (like "happi" appearing in "happy," "happiness," "unhappy")
A fixed vocabulary of 30,000 subwords can represent virtually any text in a language, while a word-level tokenizer might need millions of entries and still fail on new words.
Key Takeaway: Subword tokenization solves the OOV problem by decomposing rare words into learnable pieces, balancing vocabulary size with coverage and enabling models to understand morphological relationships between words.