Course contentsShow
← Machine Learning and Deep Learning
Lesson 1147 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

The [CLS] Token for Sequence Classification

How BERT uses the special [CLS] token's representation for sentence-level predictions.

The [CLS] Token for Sequence Classification

What you'll learn: How BERT uses a special token at the start of every input to create sentence-level predictions.

The Special [CLS] Token

When BERT processes text, it always adds a special [CLS] token at the very beginning of the input sequence. While this token has no inherent meaning in natural language, it serves a crucial architectural purpose: it aggregates information about the entire sentence.

Why [CLS] Matters

Remember that BERT uses bidirectional attention, meaning every token can "see" every other token. As information flows through BERT's 12 or 24 layers, the [CLS] token's representation becomes a rich summary of the entire input sequence.

Think of [CLS] as a class representative who attends every meeting (attention layer) and takes notes about what everyone says. By the final layer, this representative has a comprehensive understanding of the whole group's discussion.

Using [CLS] for Classification

When you want BERT to make a sentence-level prediction (like sentiment analysis, spam detection, or next sentence prediction), you:

  1. Feed your text through BERT with [CLS] at the start
  2. Extract the final hidden state of the [CLS] token
  3. Pass this representation through a classification head (typically a simple linear layer)

The [CLS] representation is a fixed-size vector (768 dimensions in BERT-base) that encodes the meaning of your entire input, regardless of sentence length.

Why Not Average All Tokens?

You could average all token representations, but [CLS] is explicitly trained during pretraining to serve as a sequence summary (especially through the Next Sentence Prediction task). This gives it an optimized role as the sentence representation.

Key Takeaway: The [CLS] token's final hidden state serves as BERT's learned representation of the entire input sequence, making it the natural choice for sentence-level classification tasks.