Special Tokens: BOS, EOS, PAD, and UNK
You'll learn how special tokens act as control signals to help models understand where sequences begin, end, and handle edge cases.
What Are Special Tokens?
When processing text, models need more than just words—they need markers that provide structural information. Special tokens are reserved symbols added to your vocabulary that serve specific purposes rather than representing actual content.
Think of special tokens like punctuation marks in written language, but for machines. Just as a period signals the end of a sentence to you, an EOS token signals the end of a sequence to a model.
The Four Essential Special Tokens
BOS (Beginning of Sequence)
Marks where a sequence starts. Some models use this to initialize their internal state for generation or to signal context boundaries.
EOS (End of Sequence)
Indicates where a sequence ends. Critical for autoregressive models during generation—it tells them when to stop producing tokens.
PAD (Padding)
When batching sequences of different lengths, shorter sequences need padding to match the longest one. PAD tokens fill these gaps and are typically ignored during training through attention masking.
UNK (Unknown)
Represents words that weren't in the training vocabulary. When you encounter a rare word like "pneumonoultramicroscopicsilicovolcanoconiosis," it becomes UNK if your tokenizer hasn't seen it before.
Why This Matters
Without these tokens, models couldn't handle variable-length sequences efficiently, wouldn't know when to stop generating text, and would crash on unfamiliar words. They're the invisible infrastructure that makes modern NLP possible.
Key Takeaway: Special tokens (BOS, EOS, PAD, UNK) provide essential control signals that help models understand sequence boundaries, enable efficient batching, and gracefully handle unknown words.