Special Tokens and Their Roles
You'll learn: How special tokens like [PAD], [CLS], [SEP], [MASK], and [UNK] act as control signals that guide transformer models during training and inference.
What Are Special Tokens?
Special tokens are reserved vocabulary entries that serve specific control functions rather than representing actual text. Think of them as punctuation marks or stage directions in a play—they don't carry the story's content, but they tell the actors (models) how to perform.
Common Special Tokens and Their Roles
[PAD] – Padding Token
Used to make all sequences in a batch the same length. Since transformers process batches efficiently, shorter sequences get padded with [PAD] tokens. The model learns to ignore these during attention.
[CLS] – Classification Token
Placed at the start of sequences in BERT-style models. Its final hidden state aggregates information from the entire sequence, making it ideal for classification tasks.
[SEP] – Separator Token
Marks boundaries between different segments (e.g., separating question from context in Q&A tasks, or sentence pairs in entailment tasks).
[MASK] – Mask Token
Central to BERT's pretraining. Replaces random tokens during training, forcing the model to predict the original word from context—teaching bidirectional understanding.
[UNK] – Unknown Token
Represents out-of-vocabulary words the tokenizer doesn't recognize. Acts as a fallback for rare or unseen words.
Custom Special Tokens
You can define domain-specific markers like <speaker1> or <code> to structure specialized tasks.
How They're Used
During training, these tokens guide learning objectives (masking for MLM, classification from [CLS]). During inference, they structure input format and control how predictions are made or decoded.
Key Takeaway: Special tokens are the "grammar" of transformer communication—they don't carry meaning themselves, but they structure how models read, learn from, and generate sequences.