Vocabulary Size Selection
What you'll learn: How to choose the right vocabulary size for your tokenizer and understand its impact on model behavior.
Why Vocabulary Size Matters
When building or selecting a tokenizer, one of the most critical decisions is how many unique tokens to include in your vocabulary. This single parameter creates ripple effects across your entire model's performance, efficiency, and capabilities.
Think of vocabulary size like choosing how many words to include in a dictionary. A tiny pocket dictionary (small vocab) is quick to search but lacks nuance. An unabridged encyclopedia (large vocab) captures everything but becomes unwieldy.
The Three-Way Trade-off
Model Performance
- Smaller vocabularies (1K-10K tokens) force the tokenizer to break words into many pieces, creating longer sequences but simpler, more generalized representations
- Larger vocabularies (50K-100K+ tokens) keep words more intact, creating shorter sequences with richer per-token meaning
Training Speed and Memory
- Each vocabulary item needs an embedding vector in your model
- A 50K vocab with 768-dimensional embeddings = ~38M parameters just for the embedding layer
- Smaller vocabs = faster lookups, less memory, quicker training
Sequence Length
- Smaller vocabs tokenize text into MORE tokens (longer sequences)
- Longer sequences = more computational cost per example
Typical Ranges by Model Type
- Small models or limited domains: 8K-16K tokens
- Standard transformers (BERT, GPT-2): 30K-50K tokens
- Large language models (GPT-3, T5): 50K-100K tokens
- Multilingual models: 100K-250K tokens (covering many languages)
Key Takeaway: Vocabulary size is a strategic choice balancing representation quality, computational efficiency, and memory constraints—typically ranging from 8K for small models to 100K+ for large multilingual systems.