Character-Level Tokenization
You'll learn how splitting text into individual characters creates the smallest possible tokens and what trade-offs this approach brings.
The Core Idea
Character-level tokenization breaks text down to its most atomic units: individual characters. Instead of treating words or subwords as tokens, every letter, digit, punctuation mark, and space becomes its own token.
For example, the sentence "Hello!" becomes: ['H', 'e', 'l', 'l', 'o', '!']
Think of it like spelling out every word letter-by-letter, the way a young child might when first learning to read.
Benefits: Tiny Vocabulary
The biggest advantage is vocabulary size. Most languages need only 50-300 characters to represent all possible text:
- 26 lowercase letters
- 26 uppercase letters
- 10 digits
- ~20-50 punctuation and special characters
This means your model never encounters "unknown" tokens—every possible word can be represented, even misspellings, rare names, or made-up words. Memory requirements stay minimal since you're storing far fewer embeddings than word-level approaches.
Limitations: Lost Meaning
The downside is semantic sparsity. Individual characters carry almost no meaning on their own. The model must learn to compose characters into meaningful units from scratch.
- Sequences become much longer (10x-20x typical word counts)
- Training becomes slower due to processing more steps
- The model must "discover" what words are instead of starting with them
- Long-range dependencies become harder to capture
It's like trying to understand a book by reading one letter at a time—technically possible, but inefficient.
When to Consider It
Character-level tokenization works well for tasks where character patterns matter: spell-checking, DNA sequence analysis, or languages without clear word boundaries.
Key Takeaway: Character-level tokenization gives you the smallest vocabulary and handles any input, but forces models to learn meaning from scratch across much longer sequences.