Course contentsShow
← Machine Learning and Deep Learning
Lesson 1241 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Vocabulary Size Trade-offs

Balancing between vocabulary size, sequence length, and semantic granularity in tokenization design.

Vocabulary Size Trade-offs

You'll learn how choosing the right vocabulary size balances model efficiency, representation quality, and memory constraints in tokenization.

The Core Trade-off

When designing a tokenization system, you face a fundamental balancing act. A larger vocabulary (more unique tokens) means shorter sequences but requires more memory and parameters. A smaller vocabulary creates longer sequences but uses fewer embeddings.

Think of it like packing for a trip: you can bring many specialized tools (large vocabulary) making each task quick but your bag heavy, or bring fewer multi-purpose tools (small vocabulary) making your bag light but each task requiring more steps.

Three Dimensions to Balance

1. Vocabulary Size

  • Small (1,000-5,000 tokens): Very limited coverage
  • Medium (30,000-50,000 tokens): Common for word-level models
  • Large (100,000+ tokens): Subword models handle rare words better

2. Sequence Length

Smaller vocabularies break text into more pieces, creating longer sequences. Longer sequences mean more computation and memory during training and inference.

3. Semantic Granularity

Large vocabularies preserve whole words and meaning better. Small vocabularies force you to piece together meaning from fragments, potentially losing nuance but gaining flexibility with rare or new words.

The Practical Reality

Modern models typically use 30K-50K vocabulary sizes as a sweet spot. This handles common words whole (preserving meaning) while breaking rare words into meaningful subword pieces (avoiding out-of-vocabulary problems). Going much larger hits diminishing returns—you memorize rare spellings instead of learning patterns.

Key Takeaway: Vocabulary size is a three-way trade-off between memory cost, sequence length, and semantic preservation—optimal size depends on your language, domain, and computational constraints.