Tokenization and Model Architecture
What you'll learn: How your tokenization choices ripple through your entire model's design, affecting memory, speed, and performance.
The Hidden Connection
Think of tokenization as choosing the alphabet for a language. If you pick letters (character-level), you need fewer "letters" to memorize but must spell out every word. If you pick whole words (word-level), you memorize thousands of "letters" but can write faster. Your model's internal structure must adapt to whatever choice you make.
Three Critical Impacts
Embedding Table Size
Your embedding table stores one vector per token in your vocabulary. A character-level tokenizer with 256 tokens needs a tiny embedding table. A word-level tokenizer with 50,000 tokens needs 195× more memory just for embeddings. This is your model's first layer—before any neural network magic happens.
Positional Encodings and Sequence Length
More tokens per sentence = longer sequences. "Hello world" might be 2 word tokens but 11 character tokens. Longer sequences mean:
- More positional encodings to track
- Quadratically more attention computations (in standard Transformers)
- Higher memory during training
Computational Requirements
A 100-word document might become 100 word tokens or 500 subword tokens (like WordPiece). Every extra token multiplies your model's work: more embedding lookups, more attention calculations, more forward pass operations.
This is why modern models like BERT and GPT carefully balance vocabulary size (~30K-50K subwords) to avoid tiny vocabularies (too many tokens per text) or huge ones (massive embedding tables).
Key Takeaway: Tokenization isn't just preprocessing—it directly determines your model's memory footprint, sequence lengths, and computational cost, forcing architectural trade-offs before you even start training.