Course contentsShow
← Machine Learning and Deep Learning
Lesson 1259 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Special Tokens and Control Codes

Role of special tokens like [CLS], [SEP], [PAD], and how they're incorporated into tokenization.

Special Tokens and Control Codes

You'll learn how special markers like [CLS], [SEP], and [PAD] guide models through structured text processing.

What Are Special Tokens?

Special tokens are reserved markers that tokenizers insert into text to provide structural information to language models. They're not real words—they're like punctuation for machine learning, telling the model where sentences begin and end, where to ignore padding, or how to classify sequences.

Think of special tokens as traffic signals in a text highway: they don't represent content themselves, but they organize the flow of information.

Common Special Tokens

[CLS] - Classification Token

Placed at the beginning of sequences in models like BERT. The model learns to encode the entire sequence's meaning into this token's representation, making it perfect for classification tasks.

[SEP] - Separator Token

Marks boundaries between segments. In sentence-pair tasks (like question-answering), [SEP] tells the model where the first sentence ends and the second begins.

[PAD] - Padding Token

When batching sequences of different lengths, [PAD] fills shorter sequences to match the longest one. The model learns to ignore these positions during training.

[UNK] - Unknown Token

Represents words not in the vocabulary (though subword tokenization minimizes this need).

How They're Incorporated

During tokenization, special tokens are automatically inserted at appropriate positions. For example, tokenizing "Hello world" for BERT produces:

[CLS] hello world [SEP]

The tokenizer assigns each special token a unique ID in the vocabulary, and the model learns their significance through pretraining. They become part of the positional and attention calculations just like regular tokens.

Key Takeaway: Special tokens are structural markers inserted during tokenization that help models understand sequence boundaries, padding locations, and task-specific information—essential scaffolding for modern NLP architectures.