Course contentsShow
← Machine Learning and Deep Learning
Lesson 1255 of 3,53828. NLP Preprocessing and TokenizationFree lesson

WordPiece in BERT

How BERT uses WordPiece with special prefix markers and its impact on model performance.

WordPiece in BERT

You'll learn how BERT applies WordPiece tokenization with special prefix markers to handle any text efficiently.

The Core Idea

While the basic WordPiece algorithm merges frequent character pairs during training (like you learned in lesson 1254), BERT's implementation adds a twist: it marks subword pieces with a special ## prefix to distinguish word-initial tokens from continuation pieces.

When BERT tokenizes "playing", it might split it as:

  • play (word start, no prefix)
  • ##ing (continuation, with ## prefix)

This marker tells the model "this piece continues the previous word" rather than starting a new one. Without it, the model couldn't distinguish between the word "playing" and the phrase "play ing" (two separate words).

Why This Matters for BERT

BERT uses around 30,000 WordPiece tokens in its vocabulary. The ## prefix system allows it to:

  1. Balance vocabulary size: Common words stay whole ("the", "is"), while rare words break into meaningful pieces
  2. Preserve word boundaries: The model learns different representations for word starts vs. continuations
  3. Handle any input: Unknown words decompose into known subwords, eliminating the out-of-vocabulary problem
  4. Improve performance: Word boundary information helps BERT understand linguistic structure better than algorithms without positional markers

For example, "unhappiness" might become ["un", "##hap", "##pi", "##ness"], where each piece is learnable and the prefixes signal internal word structure.

Key Takeaway: BERT's WordPiece with ## prefix markers balances vocabulary efficiency with linguistic awareness, enabling the model to process any text while preserving crucial word boundary information that improves downstream task performance.