Course contentsShow
← Machine Learning and Deep Learning
Lesson 1239 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Word-Level Tokenization

Splitting text by whitespace and punctuation: intuitive approach but leads to massive vocabularies and OOV problems.

Word-Level Tokenization

What you'll learn: How splitting text into individual words creates an intuitive but problematic tokenization approach.

The Core Idea

Word-level tokenization breaks text into tokens by splitting on whitespace and punctuation. It's the most intuitive approach—just like how we naturally think of sentences as sequences of words.

For example:

  • Input: "Hello, world! How are you?"
  • Tokens: ["Hello", ",", "world", "!", "How", "are", "you", "?"]

Each unique word becomes a separate token in your vocabulary.

The Appeal

This method mirrors human language understanding. We read and process text word-by-word, so it feels natural to teach machines the same way. Implementation is straightforward—just split on spaces and separate punctuation.

The Problem: Vocabulary Explosion

Here's where word-level tokenization breaks down:

Massive vocabularies: English alone has hundreds of thousands of words. Add proper nouns, technical terms, and typos, and your vocabulary explodes to millions of entries. Each needs its own embedding vector, consuming enormous memory.

Out-of-vocabulary (OOV) nightmares: Encounter a word not in your training data? The model has no representation for it. You're forced to use an <UNK> (unknown) token, losing all semantic information. Rare words, misspellings, and domain-specific terms become black holes.

The Analogy

Imagine memorizing every possible English word and their meanings individually. You'd need massive storage, and encountering a new word would leave you completely stumped—no way to guess its meaning from familiar parts like prefixes or roots.

Key Takeaway: Word-level tokenization is intuitive but impractical—it creates unwieldy vocabularies and can't handle unseen words, motivating the need for more sophisticated tokenization strategies.