Course contentsShow
← Machine Learning and Deep Learning
Lesson 1155 of 3,53826. Pretrained Language Models: BERT FamilyFree lesson

Why NSP Was Controversial

The debate around Next Sentence Prediction's effectiveness and later models dropping it.

Why NSP Was Controversial

You'll learn why the Next Sentence Prediction (NSP) task in BERT sparked debate and was eventually dropped by many successor models.

The Core Issue

When BERT introduced Next Sentence Prediction as a pretraining objective (predicting if sentence B follows sentence A), the original paper claimed it helped with downstream tasks requiring understanding of sentence relationships. However, the research community soon questioned whether NSP actually provided meaningful benefits.

Why Researchers Had Doubts

NSP proved controversial for two main reasons:

1. The task might be too easy. The negative examples in NSP came from completely different documents, making the prediction trivial. The model could often guess correctly just by noticing topic shifts, without learning deep sentence relationships.

2. Unclear benefits. Follow-up research struggled to demonstrate that NSP actually improved performance on downstream tasks. Some experiments suggested removing NSP didn't hurt—or even improved—results on certain benchmarks.

What Happened Next

This controversy led later models to reconsider NSP:

  • RoBERTa (a BERT variant) explicitly removed NSP and showed better performance without it
  • Researchers proposed that the computational resources spent on NSP might be better used elsewhere
  • Some models replaced NSP with different objectives focused on coherence or ordering

Think of it like this: If you're learning to understand relationships between ideas, but your practice examples are "Einstein's relativity theory" followed by "How to bake cookies," you're not really learning subtle connections—you're just spotting obvious topic jumps.

Key Takeaway: NSP became controversial because its benefits were questionable while its implementation might have been too simple to capture meaningful sentence relationships, leading many subsequent models to drop it entirely.