The [CLS] Token for Sequence Classification
What you'll learn: How BERT uses a special token at the start of every input to create sentence-level predictions.
The Special [CLS] Token
When BERT processes text, it always adds a special [CLS] token at the very beginning of the input sequence. While this token has no inherent meaning in natural language, it serves a crucial architectural purpose: it aggregates information about the entire sentence.
Why [CLS] Matters
Remember that BERT uses bidirectional attention, meaning every token can "see" every other token. As information flows through BERT's 12 or 24 layers, the [CLS] token's representation becomes a rich summary of the entire input sequence.
Think of [CLS] as a class representative who attends every meeting (attention layer) and takes notes about what everyone says. By the final layer, this representative has a comprehensive understanding of the whole group's discussion.
Using [CLS] for Classification
When you want BERT to make a sentence-level prediction (like sentiment analysis, spam detection, or next sentence prediction), you:
- Feed your text through BERT with
[CLS]at the start - Extract the final hidden state of the
[CLS]token - Pass this representation through a classification head (typically a simple linear layer)
The [CLS] representation is a fixed-size vector (768 dimensions in BERT-base) that encodes the meaning of your entire input, regardless of sentence length.
Why Not Average All Tokens?
You could average all token representations, but [CLS] is explicitly trained during pretraining to serve as a sequence summary (especially through the Next Sentence Prediction task). This gives it an optimized role as the sentence representation.
Key Takeaway: The [CLS] token's final hidden state serves as BERT's learned representation of the entire input sequence, making it the natural choice for sentence-level classification tasks.