Course contentsShow
← Machine Learning and Deep Learning
Lesson 1261 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Pre-tokenization Strategies

Splitting text before subword tokenization: whitespace, punctuation, and regex-based rules.

Pre-tokenization Strategies

You'll learn: How to split raw text into initial chunks before applying subword tokenization algorithms like BPE or WordPiece.

What Is Pre-tokenization?

Before feeding text into a subword tokenizer (BPE, WordPiece, etc.), you need to break it into initial pieces. Think of pre-tokenization as the "first pass" that creates boundaries—like cutting a pizza into slices before deciding how to split those slices further.

Pre-tokenization determines where your tokenizer can and cannot merge characters. If you split on whitespace first, "hello" and "world" will never merge into "helloworld"—the space creates a hard boundary.

Common Pre-tokenization Strategies

Whitespace Splitting

The simplest approach: split on spaces, tabs, and newlines. "Hello world!" becomes ["Hello", "world!"]. Fast and intuitive, but punctuation stays attached to words.

Whitespace + Punctuation

Split on both whitespace and punctuation marks. "Hello world!" becomes ["Hello", "world", "!"]. This treats punctuation as separate tokens, which often improves vocabulary efficiency.

Regex-Based Rules

Use regular expressions to define custom splitting patterns. For example, you might:

  • Keep contractions together ("don't" stays as one unit)
  • Separate numbers from letters ("abc123" → ["abc", "123"])
  • Handle special characters differently based on context

Example pattern: Split on whitespace but keep apostrophes with words, separate other punctuation.

Why It Matters

Your pre-tokenization choice affects what subword units your tokenizer can learn. If you never split "isn't" during pre-tokenization, your BPE algorithm might merge "isn" and "t" in ways that don't generalize well to "aren't" or "wasn't".

Key Takeaway: Pre-tokenization establishes initial boundaries in text before subword algorithms run, using whitespace, punctuation rules, or regex patterns to control how text can be split downstream.