Course contentsShow
← Machine Learning and Deep Learning
Lesson 1238 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Character-Level Tokenization

Splitting text into individual characters: benefits of small vocabulary size and limitations for semantic meaning.

Character-Level Tokenization

You'll learn how splitting text into individual characters creates the smallest possible tokens and what trade-offs this approach brings.

The Core Idea

Character-level tokenization breaks text down to its most atomic units: individual characters. Instead of treating words or subwords as tokens, every letter, digit, punctuation mark, and space becomes its own token.

For example, the sentence "Hello!" becomes: ['H', 'e', 'l', 'l', 'o', '!']

Think of it like spelling out every word letter-by-letter, the way a young child might when first learning to read.

Benefits: Tiny Vocabulary

The biggest advantage is vocabulary size. Most languages need only 50-300 characters to represent all possible text:

  • 26 lowercase letters
  • 26 uppercase letters
  • 10 digits
  • ~20-50 punctuation and special characters

This means your model never encounters "unknown" tokens—every possible word can be represented, even misspellings, rare names, or made-up words. Memory requirements stay minimal since you're storing far fewer embeddings than word-level approaches.

Limitations: Lost Meaning

The downside is semantic sparsity. Individual characters carry almost no meaning on their own. The model must learn to compose characters into meaningful units from scratch.

  • Sequences become much longer (10x-20x typical word counts)
  • Training becomes slower due to processing more steps
  • The model must "discover" what words are instead of starting with them
  • Long-range dependencies become harder to capture

It's like trying to understand a book by reading one letter at a time—technically possible, but inefficient.

When to Consider It

Character-level tokenization works well for tasks where character patterns matter: spell-checking, DNA sequence analysis, or languages without clear word boundaries.

Key Takeaway: Character-level tokenization gives you the smallest vocabulary and handles any input, but forces models to learn meaning from scratch across much longer sequences.