Course contentsShow
← Machine Learning and Deep Learning
Lesson 1647 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Vocabulary Size Selection

Choosing optimal vocabulary size balancing embedding table size, sequence length, and language coverage (typically 32K-100K).

Vocabulary Size Selection

What you'll learn: How to choose the right vocabulary size for your LLM tokenizer to balance model efficiency, language coverage, and computational costs.

The Balancing Act

When building a tokenizer for a large language model, one of your most important decisions is: how many unique tokens should my vocabulary contain? Modern LLMs typically use vocabularies between 32,000 and 100,000 tokens.

Think of vocabulary size like choosing how many unique "words" your model's dictionary will hold. Too small, and you'll break everything into tiny fragments (imagine splitting "unbelievable" into "un", "be", "lie", "va", "ble"). Too large, and you're memorizing every possible word variant, wasting memory and making rare tokens impossible to learn well.

Three Key Trade-offs

1. Embedding Table Size

Each token needs an embedding vector. A 50K vocabulary with 4096-dimensional embeddings requires 200M parameters just for the embedding table. Doubling vocabulary size doubles this memory cost.

2. Sequence Length

Smaller vocabularies create longer token sequences for the same text. Since transformer attention is O(n²) in sequence length, this dramatically increases compute. Longer sequences also mean more positions to model and slower generation.

3. Language Coverage

Larger vocabularies capture more complete morphemes and common phrases as single tokens, improving the model's ability to represent language efficiently and reducing information loss from over-fragmentation.

The Sweet Spot

The 32K-100K range emerged from empirical research. Below 32K, sequences become inefficiently long. Above 100K, rare tokens don't see enough training examples to learn meaningful embeddings, and the embedding table becomes prohibitively large.

Key Takeaway: Vocabulary size selection balances three competing factors—choose 32K-100K tokens to minimize embedding table memory while keeping sequences short and maintaining good coverage of your target language(s).