Course contentsShow
← Machine Learning and Deep Learning
Lesson 1144 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Next Sentence Prediction (NSP) Task

Understanding BERT's second pretraining objective that predicts if two sentences are consecutive.

Next Sentence Prediction (NSP) Task

What you'll learn: How BERT's second pretraining objective teaches the model to understand relationships between sentence pairs.

The Core Idea

While Masked Language Modeling (MLM) helps BERT understand word-level context, many NLP tasks require understanding relationships between sentences—like question answering or natural language inference. Next Sentence Prediction (NSP) is BERT's second pretraining objective that addresses this need.

How NSP Works

During pretraining, BERT receives pairs of sentences and must predict whether sentence B actually follows sentence A in the original document:

  • 50% of the time: Sentence B truly follows sentence A (label: IsNext)
  • 50% of the time: Sentence B is randomly sampled from elsewhere (label: NotNext)

The model learns a binary classification task: "Are these sentences consecutive?"

The Architecture

BERT processes both sentences together (separated by a [SEP] token) and uses the [CLS] token's final representation to make the prediction. This special token aggregates information about the entire sentence pair.

Example:

[CLS] The cat sat on the mat. [SEP] It purred contentedly. [SEP]
→ Label: IsNext

[CLS] The cat sat on the mat. [SEP] We need more coffee. [SEP]
→ Label: NotNext

Why NSP Matters

This objective forces BERT to learn:

  • Coherence: What makes sentences logically connected
  • Topic continuity: Whether ideas flow naturally
  • Discourse relationships: How sentences relate beyond individual words

These skills directly transfer to downstream tasks like question answering (is this answer relevant to this question?) and textual entailment.

Key Takeaway: Next Sentence Prediction trains BERT to understand sentence-level relationships by predicting whether two sentences are consecutive, complementing MLM's word-level understanding and enabling better performance on tasks requiring multi-sentence reasoning.