Course contentsShow
← Machine Learning and Deep Learning
Lesson 1148 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

The [SEP] Token for Segment Separation

Using [SEP] tokens to separate sentence pairs and enable sentence relationship modeling.

The [SEP] Token for Segment Separation

What You'll Learn

You'll understand how BERT uses the special [SEP] token to mark boundaries between different text segments, enabling the model to process sentence pairs and understand their relationships.

Why We Need Segment Separation

BERT often needs to process two sentences at once to understand their relationship. For example:

  • Question answering: "What is the capital of France?" + "Paris is a beautiful city."
  • Sentence similarity: "The cat sat on the mat." + "A feline rested on the rug."
  • Natural language inference: Determining if one sentence contradicts or supports another.

The [SEP] token acts like a punctuation mark between paragraphs—it tells BERT: "The first segment ends here, and the second segment begins."

How [SEP] Works

When BERT receives input, the structure looks like this:

[CLS] Sentence A tokens [SEP] Sentence B tokens [SEP]

The [SEP] token serves two crucial purposes:

  1. Boundary marker: It explicitly separates the two text segments so the model knows where one ends and another begins
  2. Enables segment embeddings: Works with BERT's segment embeddings (Segment A vs Segment B) that you learned about in the previous lesson

Think of [SEP] as a fence between two yards—it clearly defines where one property ends and the next begins, even though they're part of the same neighborhood (input sequence).

The Complete Picture

For single sentences, you still use [SEP] to mark the end:

[CLS] Single sentence tokens [SEP]

For sentence pairs:

[CLS] First sentence [SEP] Second sentence [SEP]

This consistent structure allows BERT's bidirectional attention to learn relationships within each segment and between segments simultaneously.

Key Takeaway: The [SEP] token is BERT's segment separator that marks boundaries between text portions, enabling the model to process and understand relationships between sentence pairs through its position-based segmentation system.