Vocabulary Size Selection
What you'll learn: How to choose the right vocabulary size for your LLM tokenizer to balance model efficiency, language coverage, and computational costs.
The Balancing Act
When building a tokenizer for a large language model, one of your most important decisions is: how many unique tokens should my vocabulary contain? Modern LLMs typically use vocabularies between 32,000 and 100,000 tokens.
Think of vocabulary size like choosing how many unique "words" your model's dictionary will hold. Too small, and you'll break everything into tiny fragments (imagine splitting "unbelievable" into "un", "be", "lie", "va", "ble"). Too large, and you're memorizing every possible word variant, wasting memory and making rare tokens impossible to learn well.
Three Key Trade-offs
1. Embedding Table Size
Each token needs an embedding vector. A 50K vocabulary with 4096-dimensional embeddings requires 200M parameters just for the embedding table. Doubling vocabulary size doubles this memory cost.
2. Sequence Length
Smaller vocabularies create longer token sequences for the same text. Since transformer attention is O(n²) in sequence length, this dramatically increases compute. Longer sequences also mean more positions to model and slower generation.
3. Language Coverage
Larger vocabularies capture more complete morphemes and common phrases as single tokens, improving the model's ability to represent language efficiently and reducing information loss from over-fragmentation.
The Sweet Spot
The 32K-100K range emerged from empirical research. Below 32K, sequences become inefficiently long. Above 100K, rare tokens don't see enough training examples to learn meaningful embeddings, and the embedding table becomes prohibitively large.
Key Takeaway: Vocabulary size selection balances three competing factors—choose 32K-100K tokens to minimize embedding table memory while keeping sequences short and maintaining good coverage of your target language(s).