Course contentsShow
← Machine Learning and Deep Learning
Lesson 1237 of 3,53828. NLP Preprocessing and TokenizationFree lesson

What Is Tokenization and Why It Matters

Understanding tokenization as the bridge between raw text and numerical representations that models can process.

What Is Tokenization and Why It Matters

What you'll learn: How tokenization transforms raw text into numerical units that language models can understand and process.

The Bridge Between Text and Models

Imagine trying to explain a sentence to someone who only understands numbers. You'd need a systematic way to convert each word (or piece of text) into a number first. That's exactly what tokenization does in NLP.

Tokenization is the process of breaking down raw text into smaller units called tokens—which could be words, subwords, or even individual characters—and mapping each token to a unique numerical identifier. Without tokenization, models have no way to work with text because neural networks fundamentally operate on numbers, not letters.

Why Tokenization Matters

Throughout your learning journey, you've encountered tokenization repeatedly: BERT's WordPiece tokenization, GPT's byte-pair encoding (BPE), special tokens like [CLS] and [SEP]. These weren't arbitrary choices—they're carefully designed tokenization strategies that directly impact model performance.

Tokenization matters because:

  • It defines your vocabulary: The set of all possible tokens determines what your model can "see"
  • It handles rare words: Subword tokenization (like WordPiece or BPE) breaks unknown words into known pieces
  • It controls input size: Different tokenization schemes produce different numbers of tokens for the same text
  • It affects computational cost: More tokens mean more computation during training and inference

Think of tokenization as translating from human language to "model language"—just as a bad translation loses meaning, poor tokenization can hurt model understanding.

Key Takeaway: Tokenization is the essential preprocessing step that converts raw text into numerical tokens, forming the bridge between human-readable language and the numerical representations that transformers and other neural networks require to learn and generate language.