Course contentsShow
← Machine Learning and Deep Learning
Lesson 1263 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Subword Regularization

Using multiple tokenization segmentations during training for robustness and better generalization.

Subword Regularization

What you'll learn: How varying tokenization during training makes NLP models more robust and better at generalizing.

The Core Idea

Up until now, you've learned various tokenization methods (BPE, WordPiece, Unigram) that produce one deterministic segmentation for each word. Subword regularization flips this on its head: instead of always tokenizing a word the same way during training, you randomly choose from multiple plausible segmentations.

Think of it like learning to read handwriting. If you only ever see one person's perfect cursive, you'll struggle with messy writing. But if you practice reading the same words written by different people in different styles, you become more adaptable. Similarly, seeing "unbelievable" tokenized as ["un", "believ", "able"] sometimes and ["unbe", "liev", "able"] other times teaches your model to handle variations.

How It Works

During training (not inference), instead of picking the single most probable tokenization, you:

  1. Sample from multiple valid segmentations probabilistically
  2. Each training step might see a different breakdown of the same word
  3. The model learns that meaning isn't rigidly tied to one specific segmentation

This is particularly natural with Unigram tokenization, which already maintains probability distributions over subword sequences.

Why It Helps

  • Robustness: The model doesn't overfit to one specific tokenization pattern
  • Generalization: Handles rare words and typos better
  • Noise injection: Acts like data augmentation at the token level
  • Better embeddings: Subword representations become more flexible

Key Takeaway: Subword regularization introduces controlled randomness in tokenization during training, exposing models to multiple segmentations of the same text, which improves robustness and generalization without requiring more training data.