Span Prediction with BERT
You'll learn how BERT identifies the exact start and end positions of answer spans in text passages.
The Core Idea
When using BERT for extractive question answering (like on the SQuAD dataset), the model doesn't generate answers from scratch—it identifies which part of the given context passage contains the answer. This is called span prediction, and it works by predicting two things: where the answer starts and where it ends.
Think of it like highlighting text with a marker: you need to know where to put the marker down (start position) and where to lift it up (end position).
How BERT Fine-Tuning Enables Span Prediction
During fine-tuning for QA, BERT gets two additional linear layers on top of its encoder:
- Start position classifier: Produces a score for each token indicating how likely it is to be the start of the answer
- End position classifier: Produces a score for each token indicating how likely it is to be the end of the answer
The input to BERT is the question and context concatenated together (with special tokens separating them). BERT processes this and outputs contextualized representations for every token. Each classifier then transforms these representations into logits, which are converted to probabilities using softmax.
The token with the highest start probability becomes the predicted start, and the token with the highest end probability becomes the predicted end. The span between these positions is your answer.
Training the Span Predictors
During training, the model knows the correct start and end positions from labeled data. It learns by minimizing the cross-entropy loss between its predicted positions and the ground truth positions.
Key Takeaway: BERT fine-tuning for extractive QA adds two token-level classifiers that predict start and end positions, transforming the task into identifying which span of the context passage answers the question.