Course contentsShow
← Machine Learning and Deep Learning
Lesson 1256 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Unigram Language Model Tokenization

Understanding Unigram tokenization as a probabilistic approach with subword unit probabilities.

Unigram Language Model Tokenization

You'll learn how Unigram tokenization treats subword splitting as a probability problem, choosing the most likely segmentation.

What Makes Unigram Different

Unlike BPE (which merges characters bottom-up) or WordPiece (which maximizes likelihood greedily), Unigram starts with a large vocabulary and prunes it down based on probabilities. It's a top-down, probabilistic approach.

Think of it like choosing between different ways to cut a sandwich: "sandwich" could be split as "sand-wich", "san-dwich", or "sandw-ich". Unigram assigns each subword a probability and picks the split with the highest overall likelihood.

How It Works

  1. Start large: Begin with a huge vocabulary of all possible subword units (characters, common words, frequent fragments)
  2. Assign probabilities: Calculate how likely each subword is based on training data
  3. Find best segmentation: For any word, compute the probability of all possible ways to split it using current subwords
  4. Prune iteratively: Remove subwords that hurt overall likelihood least, keeping vocabulary manageable

For example, given "unhappiness":

  • Option A: "un" + "happiness" (probability: 0.02 × 0.01 = 0.0002)
  • Option B: "unhap" + "pi" + "ness" (probability: 0.001 × 0.05 × 0.03 = 0.0000015)

Unigram picks Option A because it's more probable.

Why This Matters

Unigram's probabilistic nature means it naturally handles the vocabulary size trade-off by keeping the most statistically useful subwords. The algorithm considers multiple segmentations simultaneously, often producing more linguistically meaningful splits than greedy approaches.

Key Takeaway: Unigram tokenization uses subword probabilities to find the most likely way to split text, pruning vocabulary from large to optimal size through statistical modeling.