Token-Level Classification Heads
What You'll Learn
You'll learn how to add specialized output layers on top of BERT embeddings to classify every single token in a sequence—essential for tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.
From Sequence to Token Labels
In the previous lesson, you saw task-specific heads for sequence classification (one label for the entire input). But many NLP tasks require labeling each token individually:
- Named Entity Recognition: Is "Apple" a company, a fruit, or neither?
- POS Tagging: Is this word a noun, verb, or adjective?
How Token-Level Heads Work
After BERT processes your input, you get contextualized embeddings for every token (not just the [CLS] token). A token-level classification head works like this:
- Extract all token embeddings from BERT's final layer (shape:
[batch_size, sequence_length, hidden_size]) - Apply a linear layer to each token embedding independently: maps from
hidden_sizetonum_labels - Get predictions for every position: each token now has scores for all possible classes (e.g., B-PER, I-ORG, O for NER)
Think of it like giving each word in a sentence its own mini-classifier—but all these classifiers share the same parameters and benefit from BERT's contextual understanding.
The linear layer typically looks like: nn.Linear(768, num_labels) for BERT-Base, where 768 is the hidden dimension and num_labels is your tag vocabulary size.
Unlike sequence classification where you only care about [CLS], here you apply classification to every token position simultaneously.
Key Takeaway: Token-level classification heads add a simple linear layer on top of all BERT token embeddings, producing independent predictions for each token—perfect for NER, POS tagging, and other word-level labeling tasks.