Course contentsShow
← Machine Learning and Deep Learning
Lesson 1240 of 3,53828. NLP Preprocessing and TokenizationFree lesson

The Out-of-Vocabulary Problem

Why word-level tokenizers fail on unseen words and how this motivated subword tokenization approaches.

The Out-of-Vocabulary Problem

What You'll Learn

You'll understand why word-level tokenizers struggle with unseen words and what motivated the development of smarter tokenization strategies.

The Core Issue

When you use word-level tokenization (lesson 1239), you build a vocabulary of all unique words in your training data. But what happens when your model encounters a word it has never seen before?

This is the out-of-vocabulary (OOV) problem: any word not in your fixed vocabulary becomes impossible to process. The model has no representation for it—no embedding, no way to understand it.

Why This Happens

Imagine you train on news articles and build a vocabulary of 50,000 words. Your model learns embeddings for "running," "runner," and "runs." Now someone inputs "runnable"—a word that wasn't in your training data. The word-level tokenizer has two bad options:

  1. Replace it with <UNK> (unknown token): All unknown words get the same representation, losing all meaning
  2. Skip it entirely: Lose information from the input

Real-World Impact

  • Misspellings: "recieve" instead of "receive" becomes <UNK>
  • Rare words: Technical terms, proper nouns, neologisms
  • Morphological variants: "unbelievably" might be OOV even if "believe" isn't
  • Domain shift: Medical model encounters legal terminology

The Motivation for Subword Tokenization

This problem directly motivated subword tokenization approaches that break words into smaller, reusable pieces. Instead of treating "runnable" as unknown, subword methods might split it into "run" + "##able"—both pieces the model knows.

Key Takeaway: Word-level tokenization fails on unseen words by mapping them all to a single unknown token, losing critical information. This fundamental limitation drove the development of subword tokenization methods that can handle any text by breaking words into learnable components.