Byte-Level BPE
What you'll learn: How GPT-2 achieves universal text coverage by applying BPE directly to UTF-8 bytes instead of characters.
The Universal Coverage Problem
Traditional BPE operates on characters — but different languages use different character sets, and some rare symbols or emojis might not be in your vocabulary at all. This creates the out-of-vocabulary problem all over again, even with subword tokenization.
Byte-Level BPE solves this by working at an even lower level: raw bytes.
How It Works
Every piece of text, in any language, can be encoded as UTF-8 bytes (values 0-255). GPT-2 treats these 256 possible byte values as the base vocabulary, then applies the standard BPE merging algorithm on top of them.
Think of it like building with LEGO: instead of starting with letter blocks (which only exist for certain alphabets), you start with 256 tiny universal pieces that can represent anything — English, Chinese, emoji, code, even binary data.
The BPE algorithm then learns common byte sequences and merges them into larger tokens, just like it would with characters. The key difference: you can always fall back to individual bytes, guaranteeing you can encode any text without unknown tokens.
Why GPT-2 Chose This
Byte-level BPE gives GPT-2 true universality. It handles:
- Multiple languages seamlessly
- Code and special symbols
- Rare unicode characters
- Future symbols not in training data
The trade-off is slightly longer sequences for non-English text (since multi-byte UTF-8 characters might not merge as efficiently), but the robustness is worth it.
Key Takeaway: Byte-level BPE operates on UTF-8 bytes (256 base tokens) instead of characters, guaranteeing universal coverage of any text while still learning efficient subword merges — this is how GPT-2 handles any input without unknown tokens.