Course contentsShow
← Machine Learning and Deep Learning
Lesson 1249 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Why Subword Tokenization?

Understanding the limitations of word-level and character-level tokenization and why subword units solve the OOV problem.

Why Subword Tokenization?

You'll learn: Why splitting text into subword units solves critical problems that character-level and word-level tokenization cannot handle alone.

The Problem with Extremes

You've already seen two tokenization approaches, but each has deal-breaking flaws:

Word-level tokenization treats every unique word as a separate token. This creates massive vocabularies and explodes when encountering new words (the OOV problem). Words like "running," "runner," and "runs" get entirely separate entries with no shared understanding.

Character-level tokenization eliminates OOV issues—every word is just a sequence of known characters. But now your model must learn to build words from scratch every time, making sequences much longer and relationships harder to capture. The word "transformer" becomes 11 separate tokens instead of one meaningful unit.

The Subword Solution

Subword tokenization finds the middle ground. It breaks text into meaningful chunks smaller than words but larger than characters. Think of it like recognizing word parts: "unhappiness" might become ["un", "happi", "ness"].

This approach gives you the best of both worlds:

  • Rare words? Break them into known subwords instead of marking as <UNK>
  • Common words? Keep them as single tokens for efficiency
  • Related words? Share subword pieces (like "happi" appearing in "happy," "happiness," "unhappy")

A fixed vocabulary of 30,000 subwords can represent virtually any text in a language, while a word-level tokenizer might need millions of entries and still fail on new words.

Key Takeaway: Subword tokenization solves the OOV problem by decomposing rare words into learnable pieces, balancing vocabulary size with coverage and enabling models to understand morphological relationships between words.