Course contentsShow
← Machine Learning and Deep Learning
Lesson 1252 of 3,53828. NLP Preprocessing and TokenizationFree lesson

BPE Training Process

Step-by-step process of learning BPE merge rules from a corpus by counting and merging frequent pairs.

BPE Training Process

You'll learn: How BPE builds its vocabulary by iteratively finding and merging the most frequent character pairs in your training data.

The Core Idea

Byte Pair Encoding (BPE) training is a bottom-up vocabulary building process. You start with individual characters as your initial vocabulary, then repeatedly find the most common pair of adjacent tokens and merge them into a single new token. This continues until you reach your desired vocabulary size.

Think of it like learning slang: at first you know individual words, but over time you notice "going to" appears together so often that it becomes "gonna" — a single unit.

Step-by-Step Process

1. Initialize with Characters

Start with every unique character in your corpus as a separate token. Your text becomes a sequence of individual characters.

2. Count All Adjacent Pairs

Scan through the entire corpus and count how many times each pair of adjacent tokens appears together.

3. Merge the Most Frequent Pair

Find the pair that occurs most often (e.g., "e" followed by "r" appears 5,000 times). Create a new token "er" and replace all occurrences of this pair in your corpus.

4. Update Vocabulary

Add the new merged token "er" to your vocabulary.

5. Repeat

Go back to step 2 with your updated corpus. Now "er" is treated as a single token, so new pairs might include "th" + "er" → "ther".

6. Stop Condition

Continue until you've performed a predetermined number of merges or reached your target vocabulary size.

Key Takeaway: BPE training learns merge rules by greedily combining the most frequent adjacent token pairs, building from characters up to common subwords and words — capturing both common words as single tokens and rare words as sensible character sequences.