Course contentsShow
← Machine Learning and Deep Learning
Lesson 1242 of 3,53828. NLP Preprocessing and TokenizationFree lesson

Special Tokens: BOS, EOS, PAD, and UNK

Understanding the role of special tokens for marking sequence boundaries, padding, and unknown words.

Special Tokens: BOS, EOS, PAD, and UNK

You'll learn how special tokens act as control signals to help models understand where sequences begin, end, and handle edge cases.

What Are Special Tokens?

When processing text, models need more than just words—they need markers that provide structural information. Special tokens are reserved symbols added to your vocabulary that serve specific purposes rather than representing actual content.

Think of special tokens like punctuation marks in written language, but for machines. Just as a period signals the end of a sentence to you, an EOS token signals the end of a sequence to a model.

The Four Essential Special Tokens

BOS (Beginning of Sequence)

Marks where a sequence starts. Some models use this to initialize their internal state for generation or to signal context boundaries.

EOS (End of Sequence)

Indicates where a sequence ends. Critical for autoregressive models during generation—it tells them when to stop producing tokens.

PAD (Padding)

When batching sequences of different lengths, shorter sequences need padding to match the longest one. PAD tokens fill these gaps and are typically ignored during training through attention masking.

UNK (Unknown)

Represents words that weren't in the training vocabulary. When you encounter a rare word like "pneumonoultramicroscopicsilicovolcanoconiosis," it becomes UNK if your tokenizer hasn't seen it before.

Why This Matters

Without these tokens, models couldn't handle variable-length sequences efficiently, wouldn't know when to stop generating text, and would crash on unfamiliar words. They're the invisible infrastructure that makes modern NLP possible.

Key Takeaway: Special tokens (BOS, EOS, PAD, UNK) provide essential control signals that help models understand sequence boundaries, enable efficient batching, and gracefully handle unknown words.