WordPiece and Unigram Tokenization
What You'll Learn
You'll discover two powerful alternatives to BPE tokenization that use different strategies for building subword vocabularies.
The Core Idea
While BPE builds vocabulary by merging the most frequent pairs, WordPiece and Unigram take different approaches to the same problem: finding optimal subword units.
WordPiece Tokenization
WordPiece, famously used in BERT, is similar to BPE but with a key difference in how it selects merges. Instead of choosing the most frequent pair, WordPiece selects the pair that maximizes the likelihood of the training data when added to the vocabulary.
Think of it like this: BPE asks "What appears together most often?" while WordPiece asks "What merge helps us model our text most effectively?"
The algorithm starts with characters and greedily adds the merge that best improves the language model's probability of the training corpus. This often leads to more linguistically meaningful subwords.
Unigram Tokenization
Unigram takes the opposite approach: it starts with a large initial vocabulary and iteratively removes tokens that hurt the overall likelihood least. It's like pruning a tree—you begin with many branches and cut away the least useful ones.
The Unigram model treats tokenization probabilistically: multiple segmentations of the same word are possible, each with a probability. During inference, it typically uses the Viterbi algorithm to find the most likely segmentation.
This top-down approach often produces more consistent subword boundaries and handles rare words gracefully.
Key Differences
- BPE: Bottom-up, frequency-based merging
- WordPiece: Bottom-up, likelihood-based merging (used in BERT)
- Unigram: Top-down, probabilistic pruning
Key Takeaway: WordPiece optimizes for language modeling likelihood during vocabulary construction, while Unigram uses a probabilistic top-down approach, both offering alternatives to BPE's simple frequency-based merging.