The Out-of-Vocabulary Problem
What You'll Learn
You'll understand why word-level tokenizers struggle with unseen words and what motivated the development of smarter tokenization strategies.
The Core Issue
When you use word-level tokenization (lesson 1239), you build a vocabulary of all unique words in your training data. But what happens when your model encounters a word it has never seen before?
This is the out-of-vocabulary (OOV) problem: any word not in your fixed vocabulary becomes impossible to process. The model has no representation for it—no embedding, no way to understand it.
Why This Happens
Imagine you train on news articles and build a vocabulary of 50,000 words. Your model learns embeddings for "running," "runner," and "runs." Now someone inputs "runnable"—a word that wasn't in your training data. The word-level tokenizer has two bad options:
- Replace it with
<UNK>(unknown token): All unknown words get the same representation, losing all meaning - Skip it entirely: Lose information from the input
Real-World Impact
- Misspellings: "recieve" instead of "receive" becomes
<UNK> - Rare words: Technical terms, proper nouns, neologisms
- Morphological variants: "unbelievably" might be OOV even if "believe" isn't
- Domain shift: Medical model encounters legal terminology
The Motivation for Subword Tokenization
This problem directly motivated subword tokenization approaches that break words into smaller, reusable pieces. Instead of treating "runnable" as unknown, subword methods might split it into "run" + "##able"—both pieces the model knows.
Key Takeaway: Word-level tokenization fails on unseen words by mapping them all to a single unknown token, losing critical information. This fundamental limitation drove the development of subword tokenization methods that can handle any text by breaking words into learnable components.