Course contentsShow
← Machine Learning and Deep Learning
Lesson 1174 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Task-Specific Heads for Classification

Add classification heads on top of BERT's [CLS] token for sentence and document-level tasks.

Task-Specific Heads for Classification

What you'll learn: How to add simple classification layers on top of BERT's pretrained encoder to solve specific tasks like sentiment analysis or topic classification.

The Core Idea

You've learned that BERT produces rich contextual embeddings for every token, including the special [CLS] token that appears at the start of every sequence. Think of the [CLS] token's output as a "summary vector" of the entire input.

A task-specific head is just a small neural network (often a single linear layer) that you attach on top of BERT to map this [CLS] representation to your specific classification problem.

Why This Matters

BERT itself doesn't "know" whether you want to classify sentiment, detect spam, or categorize news articles. It only creates contextual embeddings. The task-specific head translates BERT's rich understanding into the exact output format you need.

Analogy: BERT is like a universal language understanding engine. The task-specific head is like a specialized attachment—a "sentiment detector nozzle" or "topic classifier nozzle"—that channels BERT's understanding toward your specific goal.

How It Works

  1. Extract the [CLS] token's final hidden state from BERT's output (usually a 768-dimensional vector for BERT-Base)
  2. Pass it through a simple classifier: typically a linear layer followed by softmax
  3. During fine-tuning, you update both BERT's weights AND the head's weights together

For a binary sentiment task: [CLS] embedding (768-dim) → Linear layer (768 → 2) → Softmax → [Positive/Negative probabilities]

For multi-class topic classification: [CLS] embedding → Linear layer (768 → num_classes) → Softmax → Class probabilities

Why [CLS] Specifically?

During pretraining, BERT learned to encode sentence-level information into the [CLS] token (remember Next Sentence Prediction?). This makes it naturally suited for sentence- and document-level tasks.

Key Takeaway: Task-specific heads are lightweight classifiers added on top of BERT's [CLS] token output, transforming general-purpose embeddings into task-specific predictions through fine-tuning.