Course contentsShow
← Machine Learning and Deep Learning
Lesson 1289 of 3,53829. Core NLP TasksFree lesson

NER as Token Classification

Frame NER as a token-level classification problem where each token receives an entity label or 'O' for outside.

NER as Token Classification

What you'll learn: How to frame Named Entity Recognition as a per-token classification problem that modern neural models can solve.

The Core Idea

In NER as token classification, you treat entity recognition like labeling each word (or token) in a sentence individually. Instead of looking at the whole sentence and extracting entities as separate objects, you go token-by-token and ask: "What label does this token get?"

Each token receives one label from your tag scheme (like IOB or BIOES). For example, if you're using IOB tags, a token might be labeled:

  • B-PER (beginning of a person name)
  • I-ORG (inside an organization name)
  • O (outside any entity — just regular text)

Why This Framing Works

Think of it like color-coding words in a document. You go through with a highlighter, and for each word you decide: "Is this part of a person's name? An organization? Or nothing special?" You're making one decision per word.

This transforms NER into a sequence labeling problem where:

  • Input: A sequence of tokens
  • Output: A sequence of labels (same length)
  • Task: Classify each token position independently (though context matters)

Modern models like BERT excel at this because they create contextual representations for each token, then add a simple classification layer on top to predict the label for that position.

Example

Sentence: "Barack Obama visited Paris"

Token-level classification:

  • Barack → B-PER
  • Obama → I-PER
  • visited → O
  • Paris → B-LOC

Key Takeaway: NER as token classification treats entity recognition as predicting one label per token, making it straightforward to implement with neural architectures that produce per-token representations.