Handling Special Tokens
What you'll learn: How to design and use special tokens in LLM tokenizers to mark padding, unknown words, sequence boundaries, and system instructions.
Why Special Tokens Matter
Every tokenizer needs more than just vocabulary—it needs special markers to handle structural information. Think of these like punctuation in writing: they're not content, but they're essential for understanding structure.
Core Special Token Types
Padding Token (<pad>)
When batching sequences of different lengths, you pad shorter ones to match the longest. The <pad> token fills these empty slots and is typically ignored by the model's attention mechanism.
Unknown Token (<unk>)
When the tokenizer encounters a word or character sequence not in its vocabulary, it replaces it with <unk>. This ensures the model can still process any input, even if information is lost.
Beginning/End of Sequence (<bos>/<eos>)
These mark where sequences start and end. During training, <eos> tells the model when to stop generating. During inference, seeing <eos> in the output signals completion.
System Tokens
Modern LLMs use special tokens for instructions and role-playing:
<|system|>,<|user|>,<|assistant|>separate different speakers<|im_start|>,<|im_end|>(instant message markers) structure conversations
Design Considerations
Uniqueness: Special tokens must never collide with normal vocabulary. Using unusual character combinations like <|token|> ensures this.
Reservation: These tokens are added to the vocabulary explicitly and assigned fixed IDs, often at the beginning or end of the vocabulary range.
Model awareness: The model learns to treat these differently—padding tokens don't contribute to loss, <eos> triggers stopping conditions.
Key Takeaway: Special tokens are the scaffolding of LLM tokenization, providing structural markers that separate content from control, enable batching, and handle edge cases—all critical for training and inference.