BPE Training Process
You'll learn: How BPE builds its vocabulary by iteratively finding and merging the most frequent character pairs in your training data.
The Core Idea
Byte Pair Encoding (BPE) training is a bottom-up vocabulary building process. You start with individual characters as your initial vocabulary, then repeatedly find the most common pair of adjacent tokens and merge them into a single new token. This continues until you reach your desired vocabulary size.
Think of it like learning slang: at first you know individual words, but over time you notice "going to" appears together so often that it becomes "gonna" — a single unit.
Step-by-Step Process
1. Initialize with Characters
Start with every unique character in your corpus as a separate token. Your text becomes a sequence of individual characters.
2. Count All Adjacent Pairs
Scan through the entire corpus and count how many times each pair of adjacent tokens appears together.
3. Merge the Most Frequent Pair
Find the pair that occurs most often (e.g., "e" followed by "r" appears 5,000 times). Create a new token "er" and replace all occurrences of this pair in your corpus.
4. Update Vocabulary
Add the new merged token "er" to your vocabulary.
5. Repeat
Go back to step 2 with your updated corpus. Now "er" is treated as a single token, so new pairs might include "th" + "er" → "ther".
6. Stop Condition
Continue until you've performed a predetermined number of merges or reached your target vocabulary size.
Key Takeaway: BPE training learns merge rules by greedily combining the most frequent adjacent token pairs, building from characters up to common subwords and words — capturing both common words as single tokens and rare words as sensible character sequences.