Course contentsShow
← Machine Learning and Deep Learning
Lesson 1258 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Byte-Level BPE

How GPT-2's byte-level BPE operates directly on UTF-8 bytes for universal coverage.

Byte-Level BPE

What you'll learn: How GPT-2 achieves universal text coverage by applying BPE directly to UTF-8 bytes instead of characters.

The Universal Coverage Problem

Traditional BPE operates on characters — but different languages use different character sets, and some rare symbols or emojis might not be in your vocabulary at all. This creates the out-of-vocabulary problem all over again, even with subword tokenization.

Byte-Level BPE solves this by working at an even lower level: raw bytes.

How It Works

Every piece of text, in any language, can be encoded as UTF-8 bytes (values 0-255). GPT-2 treats these 256 possible byte values as the base vocabulary, then applies the standard BPE merging algorithm on top of them.

Think of it like building with LEGO: instead of starting with letter blocks (which only exist for certain alphabets), you start with 256 tiny universal pieces that can represent anything — English, Chinese, emoji, code, even binary data.

The BPE algorithm then learns common byte sequences and merges them into larger tokens, just like it would with characters. The key difference: you can always fall back to individual bytes, guaranteeing you can encode any text without unknown tokens.

Why GPT-2 Chose This

Byte-level BPE gives GPT-2 true universality. It handles:

  • Multiple languages seamlessly
  • Code and special symbols
  • Rare unicode characters
  • Future symbols not in training data

The trade-off is slightly longer sequences for non-English text (since multi-byte UTF-8 characters might not merge as efficiently), but the robustness is worth it.

Key Takeaway: Byte-level BPE operates on UTF-8 bytes (256 base tokens) instead of characters, guaranteeing universal coverage of any text while still learning efficient subword merges — this is how GPT-2 handles any input without unknown tokens.