NER as Token Classification
What you'll learn: How to frame Named Entity Recognition as a per-token classification problem that modern neural models can solve.
The Core Idea
In NER as token classification, you treat entity recognition like labeling each word (or token) in a sentence individually. Instead of looking at the whole sentence and extracting entities as separate objects, you go token-by-token and ask: "What label does this token get?"
Each token receives one label from your tag scheme (like IOB or BIOES). For example, if you're using IOB tags, a token might be labeled:
B-PER(beginning of a person name)I-ORG(inside an organization name)O(outside any entity — just regular text)
Why This Framing Works
Think of it like color-coding words in a document. You go through with a highlighter, and for each word you decide: "Is this part of a person's name? An organization? Or nothing special?" You're making one decision per word.
This transforms NER into a sequence labeling problem where:
- Input: A sequence of tokens
- Output: A sequence of labels (same length)
- Task: Classify each token position independently (though context matters)
Modern models like BERT excel at this because they create contextual representations for each token, then add a simple classification layer on top to predict the label for that position.
Example
Sentence: "Barack Obama visited Paris"
Token-level classification:
Barack→B-PERObama→I-PERvisited→OParis→B-LOC
Key Takeaway: NER as token classification treats entity recognition as predicting one label per token, making it straightforward to implement with neural architectures that produce per-token representations.