Sequence Classification with Transformers
You'll learn how transformers convert entire sequences of text into single vectors suitable for classification tasks.
The Challenge of Going from Sequence to Label
When you fine-tune BERT for text classification, you face a fundamental question: BERT outputs one vector per token, but you need one prediction for the entire document. How do you aggregate all those token-level representations into a single document-level vector?
The [CLS] Token Strategy
Transformers like BERT use a special [CLS] (classification) token prepended to every input sequence. During pretraining, this token learns to aggregate information from the entire sequence through the attention mechanism. Think of it as a "summary representative" that attends to all other tokens and builds a holistic understanding.
How it works:
- Input:
[CLS] This movie was amazing [SEP] - BERT processes the sequence
- Extract the final hidden state corresponding to
[CLS] - Feed this vector into a classification head (typically a simple linear layer)
- Output: class prediction
Alternative: Pooling Strategies
While [CLS] is standard, other approaches exist:
- Mean pooling: Average all token representations (excluding special tokens)
- Max pooling: Take element-wise maximum across all token vectors
- CLS + pooling hybrid: Combine both approaches
Mean pooling can be useful when you haven't pretrained with a [CLS] token or want to leverage information more democratically across all tokens.
Why This Matters
The choice between [CLS] and pooling affects how your model aggregates context. [CLS] relies on attention to gather relevant information, while pooling uses simpler statistical aggregation.
Key Takeaway: Sequence classification with transformers converts token-level representations into document-level vectors, typically using the [CLS] token's final hidden state or pooling strategies, before passing through a classification layer.