Next Sentence Prediction (NSP) Task
What you'll learn: How BERT's second pretraining objective teaches the model to understand relationships between sentence pairs.
The Core Idea
While Masked Language Modeling (MLM) helps BERT understand word-level context, many NLP tasks require understanding relationships between sentences—like question answering or natural language inference. Next Sentence Prediction (NSP) is BERT's second pretraining objective that addresses this need.
How NSP Works
During pretraining, BERT receives pairs of sentences and must predict whether sentence B actually follows sentence A in the original document:
- 50% of the time: Sentence B truly follows sentence A (label:
IsNext) - 50% of the time: Sentence B is randomly sampled from elsewhere (label:
NotNext)
The model learns a binary classification task: "Are these sentences consecutive?"
The Architecture
BERT processes both sentences together (separated by a [SEP] token) and uses the [CLS] token's final representation to make the prediction. This special token aggregates information about the entire sentence pair.
Example:
[CLS] The cat sat on the mat. [SEP] It purred contentedly. [SEP]
→ Label: IsNext
[CLS] The cat sat on the mat. [SEP] We need more coffee. [SEP]
→ Label: NotNext
Why NSP Matters
This objective forces BERT to learn:
- Coherence: What makes sentences logically connected
- Topic continuity: Whether ideas flow naturally
- Discourse relationships: How sentences relate beyond individual words
These skills directly transfer to downstream tasks like question answering (is this answer relevant to this question?) and textual entailment.
Key Takeaway: Next Sentence Prediction trains BERT to understand sentence-level relationships by predicting whether two sentences are consecutive, complementing MLM's word-level understanding and enabling better performance on tasks requiring multi-sentence reasoning.