Special Tokens and Control Codes
You'll learn how special markers like [CLS], [SEP], and [PAD] guide models through structured text processing.
What Are Special Tokens?
Special tokens are reserved markers that tokenizers insert into text to provide structural information to language models. They're not real words—they're like punctuation for machine learning, telling the model where sentences begin and end, where to ignore padding, or how to classify sequences.
Think of special tokens as traffic signals in a text highway: they don't represent content themselves, but they organize the flow of information.
Common Special Tokens
[CLS] - Classification Token
Placed at the beginning of sequences in models like BERT. The model learns to encode the entire sequence's meaning into this token's representation, making it perfect for classification tasks.
[SEP] - Separator Token
Marks boundaries between segments. In sentence-pair tasks (like question-answering), [SEP] tells the model where the first sentence ends and the second begins.
[PAD] - Padding Token
When batching sequences of different lengths, [PAD] fills shorter sequences to match the longest one. The model learns to ignore these positions during training.
[UNK] - Unknown Token
Represents words not in the vocabulary (though subword tokenization minimizes this need).
How They're Incorporated
During tokenization, special tokens are automatically inserted at appropriate positions. For example, tokenizing "Hello world" for BERT produces:
[CLS] hello world [SEP]
The tokenizer assigns each special token a unique ID in the vocabulary, and the model learns their significance through pretraining. They become part of the positional and attention calculations just like regular tokens.
Key Takeaway: Special tokens are structural markers inserted during tokenization that help models understand sequence boundaries, padding locations, and task-specific information—essential scaffolding for modern NLP architectures.