Tokenizer Training and Corpus Selection
What you'll learn: How to train a tokenizer on the right data so your LLM can handle its intended domain effectively.
Why Corpus Selection Matters
When you train a tokenizer (whether BPE, WordPiece, or Unigram), it learns its vocabulary by analyzing which sequences of characters appear most frequently in your training corpus. If you train on English Wikipedia, you'll get excellent tokens for "encyclopedia" and "mathematics," but terrible fragmentation for medical terms like "pharmacokinetics" or code like def __init__.
Think of it like teaching someone vocabulary: if they only read news articles, they won't know culinary jargon.
Representative Corpus Selection
Your tokenizer training corpus should mirror your pretraining data mix. If your LLM will train on 60% web text, 20% code, 15% books, and 5% scientific papers, your tokenizer should see similar proportions. This ensures:
- Domain-specific terms get dedicated tokens
- Token distribution matches actual usage
- No artificial inflation of sequence length in critical domains
Handling Domain-Specific Vocabulary
For specialized domains (medical, legal, scientific), you have two approaches:
- Train from scratch on a corpus heavy in your domain—but this may hurt general performance
- Augment the corpus to include 20-30% domain-specific text alongside general text, balancing specialization with versatility
Updating for New Domains
Once a tokenizer is trained and your model pretrained, changing it is costly—you must retrain embeddings and potentially the entire model. Better strategies:
- Anticipate domains during initial tokenizer training
- Add domain text to your corpus before committing to vocabulary
- Accept minor inefficiency rather than retraining from scratch
Key Takeaway: Train your tokenizer on a corpus that represents your model's future training data—domain coverage during tokenization is much cheaper than fixing it later.