Handling Whitespace and Boundaries
What you'll learn: How tokenizers mark where words begin and end using three different encoding strategies.
Why Word Boundaries Matter
When splitting text into subword tokens (like "playing" → "play" + "ing"), we need a way to tell the model where original word boundaries were. Without this, "ice cream" and "icecream" would look identical after tokenization!
Three Approaches to Encoding Boundaries
1. Prefix Markers (▁ or Ġ)
Some tokenizers mark the start of each word with a special symbol. SentencePiece uses ▁ (underscore character):
- Input: "ice cream"
- Tokens:
["▁ice", "▁cream"]
The ▁ signals "this token starts a new word."
2. Suffix Markers (##)
WordPiece (used in BERT) marks continuation pieces with ##:
- Input: "playing"
- Tokens:
["play", "##ing"]
Here ## means "attach this to the previous token—it doesn't start a new word."
3. Explicit Spaces
Character-level or byte-level tokenizers treat spaces as regular tokens:
- Input: "ice cream"
- Tokens:
["i", "c", "e", " ", "c", "r", "e", "a", "m"]
The space character itself carries the boundary information.
Why This Matters
These markers help the model understand:
- That "unhappy" should relate to "happy" (same root)
- That "New York" is two words, not one
- How to properly reconstruct original text during detokenization
Different tokenization algorithms choose different strategies based on their design goals and the languages they target.
Key Takeaway: Word boundary encoding (prefix markers, suffix markers, or explicit spaces) ensures models can distinguish between word parts and complete words, preserving critical meaning during tokenization.