Course contentsShow
← Machine Learning and Deep Learning
Lesson 1250 of 3,53828. NLP Preprocessing and TokenizationFree lesson

The Vocabulary Size Trade-off

Balancing vocabulary size against sequence length and rare word handling in tokenization design.

The Vocabulary Size Trade-off

You'll learn: How choosing vocabulary size in tokenization affects sequence length, rare word handling, and model efficiency.

The Core Balancing Act

When designing a tokenizer, you face a fundamental trade-off: small vocabulary vs. large vocabulary. This choice cascades through your entire NLP pipeline.

Small Vocabulary (e.g., 1,000 tokens)

With fewer unique tokens, you'll encounter more unknown words (remember the UNK token?). To handle text, you must break words into smaller pieces, creating longer sequences.

Analogy: Imagine describing objects with only 100 words. You'd need many words to describe "refrigerator" — maybe "cold food box machine."

Consequences:

  • Longer sequences = more computational cost
  • More attention calculations
  • Higher memory usage
  • But: smaller embedding matrix, fewer parameters

Large Vocabulary (e.g., 100,000 tokens)

More tokens mean you can represent words more completely, yielding shorter sequences. Rare words get dedicated tokens instead of being split or marked as UNK.

Consequences:

  • Shorter sequences = faster processing
  • Fewer attention operations
  • But: huge embedding matrix
  • Many rare-word embeddings poorly trained (not enough examples)

The Sweet Spot

Modern subword tokenization (which you just learned about!) aims for the middle ground: vocabularies of 30,000-50,000 tokens. This balances:

  • Reasonable sequence lengths
  • Manageable embedding size
  • Good handling of rare/unknown words through subword splitting

Key Takeaway: Vocabulary size is a trade-off between sequence length (computational cost) and vocabulary coverage (embedding matrix size and rare word quality). Subword methods help find an optimal middle ground.