Task-Specific Heads for Classification
What you'll learn: How to add simple classification layers on top of BERT's pretrained encoder to solve specific tasks like sentiment analysis or topic classification.
The Core Idea
You've learned that BERT produces rich contextual embeddings for every token, including the special [CLS] token that appears at the start of every sequence. Think of the [CLS] token's output as a "summary vector" of the entire input.
A task-specific head is just a small neural network (often a single linear layer) that you attach on top of BERT to map this [CLS] representation to your specific classification problem.
Why This Matters
BERT itself doesn't "know" whether you want to classify sentiment, detect spam, or categorize news articles. It only creates contextual embeddings. The task-specific head translates BERT's rich understanding into the exact output format you need.
Analogy: BERT is like a universal language understanding engine. The task-specific head is like a specialized attachment—a "sentiment detector nozzle" or "topic classifier nozzle"—that channels BERT's understanding toward your specific goal.
How It Works
- Extract the [CLS] token's final hidden state from BERT's output (usually a 768-dimensional vector for BERT-Base)
- Pass it through a simple classifier: typically a linear layer followed by softmax
- During fine-tuning, you update both BERT's weights AND the head's weights together
For a binary sentiment task: [CLS] embedding (768-dim) → Linear layer (768 → 2) → Softmax → [Positive/Negative probabilities]
For multi-class topic classification: [CLS] embedding → Linear layer (768 → num_classes) → Softmax → Class probabilities
Why [CLS] Specifically?
During pretraining, BERT learned to encode sentence-level information into the [CLS] token (remember Next Sentence Prediction?). This makes it naturally suited for sentence- and document-level tasks.
Key Takeaway: Task-specific heads are lightweight classifiers added on top of BERT's [CLS] token output, transforming general-purpose embeddings into task-specific predictions through fine-tuning.