Course contentsShow
← Machine Learning and Deep Learning
Lesson 1267 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Special Tokens and Their Roles

Purpose of [PAD], [CLS], [SEP], [MASK], [UNK], and custom special tokens; how they're used during training and inference.

Special Tokens and Their Roles

You'll learn: How special tokens like [PAD], [CLS], [SEP], [MASK], and [UNK] act as control signals that guide transformer models during training and inference.

What Are Special Tokens?

Special tokens are reserved vocabulary entries that serve specific control functions rather than representing actual text. Think of them as punctuation marks or stage directions in a play—they don't carry the story's content, but they tell the actors (models) how to perform.

Common Special Tokens and Their Roles

[PAD] – Padding Token

Used to make all sequences in a batch the same length. Since transformers process batches efficiently, shorter sequences get padded with [PAD] tokens. The model learns to ignore these during attention.

[CLS] – Classification Token

Placed at the start of sequences in BERT-style models. Its final hidden state aggregates information from the entire sequence, making it ideal for classification tasks.

[SEP] – Separator Token

Marks boundaries between different segments (e.g., separating question from context in Q&A tasks, or sentence pairs in entailment tasks).

[MASK] – Mask Token

Central to BERT's pretraining. Replaces random tokens during training, forcing the model to predict the original word from context—teaching bidirectional understanding.

[UNK] – Unknown Token

Represents out-of-vocabulary words the tokenizer doesn't recognize. Acts as a fallback for rare or unseen words.

Custom Special Tokens

You can define domain-specific markers like <speaker1> or <code> to structure specialized tasks.

How They're Used

During training, these tokens guide learning objectives (masking for MLM, classification from [CLS]). During inference, they structure input format and control how predictions are made or decoded.

Key Takeaway: Special tokens are the "grammar" of transformer communication—they don't carry meaning themselves, but they structure how models read, learn from, and generate sequences.