Course contentsShow
← Machine Learning and Deep Learning
Lesson 1648 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Handling Special Tokens

Designing special tokens for padding, unknown words, beginning/end of sequence, and system instructions in LLM tokenizers.

Handling Special Tokens

What you'll learn: How to design and use special tokens in LLM tokenizers to mark padding, unknown words, sequence boundaries, and system instructions.

Why Special Tokens Matter

Every tokenizer needs more than just vocabulary—it needs special markers to handle structural information. Think of these like punctuation in writing: they're not content, but they're essential for understanding structure.

Core Special Token Types

Padding Token (<pad>)

When batching sequences of different lengths, you pad shorter ones to match the longest. The <pad> token fills these empty slots and is typically ignored by the model's attention mechanism.

Unknown Token (<unk>)

When the tokenizer encounters a word or character sequence not in its vocabulary, it replaces it with <unk>. This ensures the model can still process any input, even if information is lost.

Beginning/End of Sequence (<bos>/<eos>)

These mark where sequences start and end. During training, <eos> tells the model when to stop generating. During inference, seeing <eos> in the output signals completion.

System Tokens

Modern LLMs use special tokens for instructions and role-playing:

  • <|system|>, <|user|>, <|assistant|> separate different speakers
  • <|im_start|>, <|im_end|> (instant message markers) structure conversations

Design Considerations

Uniqueness: Special tokens must never collide with normal vocabulary. Using unusual character combinations like <|token|> ensures this.

Reservation: These tokens are added to the vocabulary explicitly and assigned fixed IDs, often at the beginning or end of the vocabulary range.

Model awareness: The model learns to treat these differently—padding tokens don't contribute to loss, <eos> triggers stopping conditions.

Key Takeaway: Special tokens are the scaffolding of LLM tokenization, providing structural markers that separate content from control, enable batching, and handle edge cases—all critical for training and inference.