Course contentsShow
← Machine Learning and Deep Learning
Lesson 1260 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Handling Whitespace and Boundaries

Different approaches to encoding word boundaries: prefix markers, suffix markers, and explicit spaces.

Handling Whitespace and Boundaries

What you'll learn: How tokenizers mark where words begin and end using three different encoding strategies.

Why Word Boundaries Matter

When splitting text into subword tokens (like "playing" → "play" + "ing"), we need a way to tell the model where original word boundaries were. Without this, "ice cream" and "icecream" would look identical after tokenization!

Three Approaches to Encoding Boundaries

1. Prefix Markers (▁ or Ġ)

Some tokenizers mark the start of each word with a special symbol. SentencePiece uses ▁ (underscore character):

  • Input: "ice cream"
  • Tokens: ["▁ice", "▁cream"]

The ▁ signals "this token starts a new word."

2. Suffix Markers (##)

WordPiece (used in BERT) marks continuation pieces with ##:

  • Input: "playing"
  • Tokens: ["play", "##ing"]

Here ## means "attach this to the previous token—it doesn't start a new word."

3. Explicit Spaces

Character-level or byte-level tokenizers treat spaces as regular tokens:

  • Input: "ice cream"
  • Tokens: ["i", "c", "e", " ", "c", "r", "e", "a", "m"]

The space character itself carries the boundary information.

Why This Matters

These markers help the model understand:

  • That "unhappy" should relate to "happy" (same root)
  • That "New York" is two words, not one
  • How to properly reconstruct original text during detokenization

Different tokenization algorithms choose different strategies based on their design goals and the languages they target.

Key Takeaway: Word boundary encoding (prefix markers, suffix markers, or explicit spaces) ensures models can distinguish between word parts and complete words, preserving critical meaning during tokenization.