BERT's WordPiece Tokenization
What You'll Learn
You'll understand how BERT breaks text into subword pieces to efficiently handle vocabulary while managing rare and unknown words.
Why Subword Tokenization?
Imagine you're learning a new language with a limited dictionary. Instead of memorizing every possible word (impossible!), you learn common word parts that can combine to form meanings. That's exactly what WordPiece tokenization does.
Traditional word-level tokenization creates huge vocabularies and struggles with rare words, misspellings, or new terms. Character-level tokenization is too granular and loses semantic meaning. WordPiece sits perfectly in the middle.
How WordPiece Works
WordPiece breaks words into frequent subword units. Common words stay whole, while rare words split into meaningful pieces:
"playing"→["play", "##ing"]"unhappiness"→["un", "##happiness"]or["un", "##hap", "##pi", "##ness"]"Transformer"→["Trans", "##former"]
The ## prefix indicates a subword that continues from a previous piece (not a word start).
The Benefits
- Fixed vocabulary size: BERT uses ~30,000 WordPiece tokens instead of millions of possible words
- Handles rare words: Even if you've never seen "antidisestablishmentarianism," you can break it into known pieces
- No unknown tokens: Every word can be represented, even if split into characters as a last resort
- Language efficiency: Captures morphological patterns (prefixes, suffixes, roots)
For example, if BERT learned "##ing", it recognizes this pattern across "running," "jumping," and "transforming" without treating each as completely separate.
Key Takeaway: WordPiece tokenization gives BERT a compact, efficient vocabulary that handles any input by breaking rare words into learnable subword units, eliminating unknown token problems while keeping common words intact.