Byte-Level vs Character-Level Tokenization
What you'll learn: How byte-level and character-level tokenization differ in handling text and their computational trade-offs for LLMs.
Core Idea
When building tokenizers for language models, we have a fundamental choice: should we split text into characters (like 'a', 'b', 'c') or into bytes (the raw 0-255 values that represent any digital text)?
Character-level tokenization treats each Unicode character as a token. The alphabet 'a' is one token, the emoji '😀' is another. Simple and intuitive, but Unicode has over 100,000 characters, leading to a massive vocabulary size.
Byte-level tokenization treats text as a sequence of bytes (256 possible values). Every character, emoji, or symbol ultimately maps to 1-4 bytes. This gives you a fixed, tiny vocabulary of just 256 tokens that can represent any text in any language or encoding.
The Trade-offs
Think of it like building blocks: character-level gives you pre-made Lego sets (one piece = one character), while byte-level gives you individual bricks (multiple bricks = one character).
Byte-level advantages:
- Universal: handles any language, emoji, rare symbols, or even corrupted text
- Tiny vocabulary (256 vs 100,000+)
- No "unknown token" problems
Byte-level challenges:
- Longer sequences (more bytes per word means more tokens to process)
- Model must learn character composition from scratch
- Higher computational cost during training and inference
Character-level is more human-intuitive but fragile with unexpected inputs. Modern LLMs like GPT often use byte-level approaches (via BPE on bytes) for robustness.
Key Takeaway: Byte-level tokenization trades sequence length for universal coverage and a minimal vocabulary, while character-level is more compact but struggles with vocabulary explosion and rare symbols.