Course contentsShow
← Machine Learning and Deep Learning
Lesson 1175 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Token-Level Classification Heads

Design output layers for token classification tasks like NER and POS tagging using BERT embeddings.

Token-Level Classification Heads

What You'll Learn

You'll learn how to add specialized output layers on top of BERT embeddings to classify every single token in a sequence—essential for tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.

From Sequence to Token Labels

In the previous lesson, you saw task-specific heads for sequence classification (one label for the entire input). But many NLP tasks require labeling each token individually:

  • Named Entity Recognition: Is "Apple" a company, a fruit, or neither?
  • POS Tagging: Is this word a noun, verb, or adjective?

How Token-Level Heads Work

After BERT processes your input, you get contextualized embeddings for every token (not just the [CLS] token). A token-level classification head works like this:

  1. Extract all token embeddings from BERT's final layer (shape: [batch_size, sequence_length, hidden_size])
  2. Apply a linear layer to each token embedding independently: maps from hidden_size to num_labels
  3. Get predictions for every position: each token now has scores for all possible classes (e.g., B-PER, I-ORG, O for NER)

Think of it like giving each word in a sentence its own mini-classifier—but all these classifiers share the same parameters and benefit from BERT's contextual understanding.

The linear layer typically looks like: nn.Linear(768, num_labels) for BERT-Base, where 768 is the hidden dimension and num_labels is your tag vocabulary size.

Unlike sequence classification where you only care about [CLS], here you apply classification to every token position simultaneously.

Key Takeaway: Token-level classification heads add a simple linear layer on top of all BERT token embeddings, producing independent predictions for each token—perfect for NER, POS tagging, and other word-level labeling tasks.