Course contentsShow
← Machine Learning and Deep Learning
Lesson 1299 of 3,53829. Core NLP TasksFree lesson

SQuAD Dataset and Benchmarks

The Stanford Question Answering Dataset structure, metrics, and its role in driving QA research.

SQuAD Dataset and Benchmarks

You'll learn what makes SQuAD the gold standard dataset for training and evaluating extractive question answering systems.

What is SQuAD?

The Stanford Question Answering Dataset (SQuAD) is a large-scale reading comprehension dataset specifically designed for extractive QA. Released in 2016, it revolutionized how we build and evaluate question answering models by providing a standardized benchmark that lets researchers compare approaches fairly.

Dataset Structure

SQuAD consists of:

  • Passages: Wikipedia paragraphs serving as context
  • Questions: Human-generated questions about each passage
  • Answers: Text spans extracted directly from the passage (extractive answers)
  • Answer positions: Character-level start and end indices marking where answers appear

Think of it like a reading comprehension test: you're given a paragraph, asked questions about it, and must highlight the exact text that answers each question.

Key Metrics

SQuAD introduced two primary evaluation metrics:

  • Exact Match (EM): Binary score—does your predicted answer exactly match any ground truth answer? (0% or 100% per question)
  • F1 Score: Token-level overlap between predicted and ground truth answers, measuring partial credit

SQuAD's Evolution

SQuAD 1.1: All questions have answers in the passage SQuAD 2.0: Added ~50,000 "unanswerable" questions, forcing models to determine when no answer exists—making the task more realistic

Impact on Research

SQuAD became the primary benchmark that drove extractive QA forward. It enabled direct comparison between models like BiLSTM-CRF approaches and modern Transformer-based systems (like BERT), showing clear performance improvements as architectures evolved.

Key Takeaway: SQuAD provides standardized passages, questions, and answer spans with Exact Match and F1 metrics, establishing the benchmark that propelled extractive question answering research forward.