Handling Unknown Tokens and OOV Words
What you'll learn: How to deal with words your tokenizer has never seen before, and why subword methods help but don't completely solve the problem.
The Problem: Words Outside Your Vocabulary
Even with a well-trained tokenizer, you'll encounter out-of-vocabulary (OOV) words—terms that didn't appear during tokenizer training. These might be new slang, typos, rare names, or domain-specific jargon. When your tokenizer can't recognize a word, it needs a fallback strategy.
Strategy 1: The UNK Token
The simplest approach uses a special <UNK> (unknown) token. Any unrecognized word gets mapped to this single token. Think of it like a "miscellaneous" bin where everything unfamiliar gets tossed together.
The downside? All unknown words become identical to your model. "pneumonoultramicroscopicsilicovolcanoconiosis" and "xylophone" would both become <UNK>, losing all information about their actual characters or structure.
Strategy 2: Subword Tokenization to the Rescue
This is where BPE, WordPiece, and Unigram tokenization shine. Instead of treating a rare word as completely unknown, they break it into smaller, known pieces:
- Unknown word: "unsubscribe"
- Subword tokens:
["un", "##subscribe"]or["_un", "sub", "scribe"]
Even if "unsubscribe" wasn't in the training data, its components likely were. The model can still extract meaningful information from the parts.
Why UNK Still Exists
Subword tokenization dramatically reduces OOV problems but doesn't eliminate them entirely. Extreme typos, emoji sequences, or rare Unicode characters might still produce <UNK> tokens. However, the frequency drops from potentially thousands of unknowns to just a handful.
Key Takeaway: Subword tokenization breaks unknown words into recognizable pieces, reducing reliance on <UNK> tokens and preserving more linguistic information than word-level tokenization ever could.