Winograd Schema and Coreference
What you'll learn: How Winograd Schema challenges test whether language models can resolve pronouns using real-world knowledge and commonsense reasoning, not just linguistic patterns.
The Core Challenge
Imagine reading: "The trophy doesn't fit in the suitcase because it is too big." What does "it" refer to—the trophy or the suitcase? Now consider: "The trophy doesn't fit in the suitcase because it is too small." Suddenly "it" refers to the suitcase!
The Winograd Schema Challenge (WSC) tests exactly this: pronoun resolution that requires understanding the world, not just grammar. Named after AI researcher Terry Winograd, these tasks present sentence pairs where a single word change flips which noun a pronoun refers to. Language models can't rely on syntactic position or word co-occurrence—they must "know" that trophies being too big (not too small) causes fitting problems.
Why It Matters for LLM Evaluation
Unlike benchmarks that measure pattern matching or statistical correlations (covered in earlier lessons like MMLU or HellaSwag), Winograd Schema specifically targets:
- Commonsense reasoning: Understanding physical constraints, causality, social norms
- World knowledge: Knowing properties of objects, typical behaviors
- Disambiguation under uncertainty: Choosing between plausible referents
Models that score well here demonstrate genuine understanding beyond memorizing training data patterns. Poor performance reveals brittleness in reasoning—the model might excel at perplexity (Lesson 3137) yet fail these carefully crafted examples.
Evaluation Design
Winograd questions typically present binary choices after minimal context. The model must select the correct referent for an ambiguous pronoun. Success requires integrating semantic understanding with the specific scenario, making this a hard test even for modern LLMs evaluated on benchmarks like ARC or BBH.
Key Takeaway: Winograd Schema evaluation probes whether LLMs truly understand pronoun references through commonsense reasoning, not statistical shortcuts—a critical test of real-world language comprehension.