Unigram Language Model Tokenization
You'll learn how Unigram tokenization treats subword splitting as a probability problem, choosing the most likely segmentation.
What Makes Unigram Different
Unlike BPE (which merges characters bottom-up) or WordPiece (which maximizes likelihood greedily), Unigram starts with a large vocabulary and prunes it down based on probabilities. It's a top-down, probabilistic approach.
Think of it like choosing between different ways to cut a sandwich: "sandwich" could be split as "sand-wich", "san-dwich", or "sandw-ich". Unigram assigns each subword a probability and picks the split with the highest overall likelihood.
How It Works
- Start large: Begin with a huge vocabulary of all possible subword units (characters, common words, frequent fragments)
- Assign probabilities: Calculate how likely each subword is based on training data
- Find best segmentation: For any word, compute the probability of all possible ways to split it using current subwords
- Prune iteratively: Remove subwords that hurt overall likelihood least, keeping vocabulary manageable
For example, given "unhappiness":
- Option A: "un" + "happiness" (probability: 0.02 × 0.01 = 0.0002)
- Option B: "unhap" + "pi" + "ness" (probability: 0.001 × 0.05 × 0.03 = 0.0000015)
Unigram picks Option A because it's more probable.
Why This Matters
Unigram's probabilistic nature means it naturally handles the vocabulary size trade-off by keeping the most statistically useful subwords. The algorithm considers multiple segmentations simultaneously, often producing more linguistically meaningful splits than greedy approaches.
Key Takeaway: Unigram tokenization uses subword probabilities to find the most likely way to split text, pruning vocabulary from large to optimal size through statistical modeling.