Pre-tokenization Strategies
You'll learn: How to split raw text into initial chunks before applying subword tokenization algorithms like BPE or WordPiece.
What Is Pre-tokenization?
Before feeding text into a subword tokenizer (BPE, WordPiece, etc.), you need to break it into initial pieces. Think of pre-tokenization as the "first pass" that creates boundaries—like cutting a pizza into slices before deciding how to split those slices further.
Pre-tokenization determines where your tokenizer can and cannot merge characters. If you split on whitespace first, "hello" and "world" will never merge into "helloworld"—the space creates a hard boundary.
Common Pre-tokenization Strategies
Whitespace Splitting
The simplest approach: split on spaces, tabs, and newlines. "Hello world!" becomes ["Hello", "world!"]. Fast and intuitive, but punctuation stays attached to words.
Whitespace + Punctuation
Split on both whitespace and punctuation marks. "Hello world!" becomes ["Hello", "world", "!"]. This treats punctuation as separate tokens, which often improves vocabulary efficiency.
Regex-Based Rules
Use regular expressions to define custom splitting patterns. For example, you might:
- Keep contractions together ("don't" stays as one unit)
- Separate numbers from letters ("abc123" →
["abc", "123"]) - Handle special characters differently based on context
Example pattern: Split on whitespace but keep apostrophes with words, separate other punctuation.
Why It Matters
Your pre-tokenization choice affects what subword units your tokenizer can learn. If you never split "isn't" during pre-tokenization, your BPE algorithm might merge "isn" and "t" in ways that don't generalize well to "aren't" or "wasn't".
Key Takeaway: Pre-tokenization establishes initial boundaries in text before subword algorithms run, using whitespace, punctuation rules, or regex patterns to control how text can be split downstream.