Course contentsShow
← Machine Learning and Deep Learning
Lesson 1266 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Vocabulary Size Selection

How vocabulary size affects model performance, training speed, and memory; typical ranges for different model types.

Vocabulary Size Selection

What you'll learn: How to choose the right vocabulary size for your tokenizer and understand its impact on model behavior.

Why Vocabulary Size Matters

When building or selecting a tokenizer, one of the most critical decisions is how many unique tokens to include in your vocabulary. This single parameter creates ripple effects across your entire model's performance, efficiency, and capabilities.

Think of vocabulary size like choosing how many words to include in a dictionary. A tiny pocket dictionary (small vocab) is quick to search but lacks nuance. An unabridged encyclopedia (large vocab) captures everything but becomes unwieldy.

The Three-Way Trade-off

Model Performance

  • Smaller vocabularies (1K-10K tokens) force the tokenizer to break words into many pieces, creating longer sequences but simpler, more generalized representations
  • Larger vocabularies (50K-100K+ tokens) keep words more intact, creating shorter sequences with richer per-token meaning

Training Speed and Memory

  • Each vocabulary item needs an embedding vector in your model
  • A 50K vocab with 768-dimensional embeddings = ~38M parameters just for the embedding layer
  • Smaller vocabs = faster lookups, less memory, quicker training

Sequence Length

  • Smaller vocabs tokenize text into MORE tokens (longer sequences)
  • Longer sequences = more computational cost per example

Typical Ranges by Model Type

  • Small models or limited domains: 8K-16K tokens
  • Standard transformers (BERT, GPT-2): 30K-50K tokens
  • Large language models (GPT-3, T5): 50K-100K tokens
  • Multilingual models: 100K-250K tokens (covering many languages)

Key Takeaway: Vocabulary size is a strategic choice balancing representation quality, computational efficiency, and memory constraints—typically ranging from 8K for small models to 100K+ for large multilingual systems.