WordPiece Algorithm
You'll learn how WordPiece tokenization builds subword vocabularies using a likelihood-based approach, and why this differs from BPE's frequency-driven strategy.
Core Idea
WordPiece is another subword tokenization algorithm that, like BPE, breaks words into smaller pieces to handle the vocabulary size vs. coverage trade-off. However, while BPE merges the most frequent pair of symbols, WordPiece merges based on likelihood — specifically, which merge maximally increases the likelihood of the training data.
Think of it this way: BPE asks "what pair appears most often?" WordPiece asks "which merge makes our vocabulary best explain the language we're seeing?"
How WordPiece Differs from BPE
BPE's Approach
- Merges the most frequently occurring adjacent pair
- Simple frequency counting drives decisions
- Example: if "th" appears 500 times, merge it
WordPiece's Approach
- Merges the pair that maximizes the likelihood (probability) of the training corpus
- Computed as:
likelihood = frequency_of_pair / (frequency_of_first × frequency_of_second) - This ratio tells us: does this pair appear more often than random chance would predict?
The Intuition
If "t" appears 1000 times and "h" appears 800 times independently, but "th" appears together 600 times, that's statistically significant — they belong together. WordPiece captures this statistical dependency, not just raw frequency.
This makes WordPiece slightly more linguistically motivated: it identifies subword units that are genuinely meaningful together, rather than just common.
Practical Impact
WordPiece was originally developed for Google's speech recognition systems and later used in BERT. The likelihood-based merging tends to create subword units that better capture morphological patterns (prefixes, suffixes, roots) because these co-occur more predictably than chance.
Key Takeaway: WordPiece uses likelihood-based merging instead of frequency-based merging, selecting subword combinations that are statistically meaningful rather than merely common, resulting in vocabularies that better capture linguistic structure.