Course contentsShow
← Machine Learning and Deep Learning
Lesson 1243 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Tokenization and Model Architecture

How tokenization choices affect embedding tables, positional encodings, and computational requirements.

Tokenization and Model Architecture

What you'll learn: How your tokenization choices ripple through your entire model's design, affecting memory, speed, and performance.

The Hidden Connection

Think of tokenization as choosing the alphabet for a language. If you pick letters (character-level), you need fewer "letters" to memorize but must spell out every word. If you pick whole words (word-level), you memorize thousands of "letters" but can write faster. Your model's internal structure must adapt to whatever choice you make.

Three Critical Impacts

Embedding Table Size

Your embedding table stores one vector per token in your vocabulary. A character-level tokenizer with 256 tokens needs a tiny embedding table. A word-level tokenizer with 50,000 tokens needs 195× more memory just for embeddings. This is your model's first layer—before any neural network magic happens.

Positional Encodings and Sequence Length

More tokens per sentence = longer sequences. "Hello world" might be 2 word tokens but 11 character tokens. Longer sequences mean:

  • More positional encodings to track
  • Quadratically more attention computations (in standard Transformers)
  • Higher memory during training

Computational Requirements

A 100-word document might become 100 word tokens or 500 subword tokens (like WordPiece). Every extra token multiplies your model's work: more embedding lookups, more attention calculations, more forward pass operations.

This is why modern models like BERT and GPT carefully balance vocabulary size (~30K-50K subwords) to avoid tiny vocabularies (too many tokens per text) or huge ones (massive embedding tables).

Key Takeaway: Tokenization isn't just preprocessing—it directly determines your model's memory footprint, sequence lengths, and computational cost, forcing architectural trade-offs before you even start training.