Course contentsShow
← Machine Learning and Deep Learning
Lesson 3149 of 3,53869. LLM EvaluationFree lesson

HellaSwag and Commonsense Reasoning

Testing commonsense reasoning through sentence completion tasks that require physical and social intuition.

HellaSwag and Commonsense Reasoning

You'll learn how HellaSwag tests whether language models can complete everyday scenarios with common sense, revealing gaps that raw perplexity can't detect.

What Is HellaSwag?

HellaSwag (from "Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations") is a benchmark that evaluates commonsense reasoning through sentence completion tasks. Unlike MMLU's factual knowledge tests, HellaSwag focuses on whether a model understands physical and social intuitions about the real world.

Each item presents a context describing an everyday activity—like making a sandwich, playing basketball, or brushing teeth—followed by four possible continuations. Only one logically follows from common sense; the others are adversarially generated to be plausible at the surface level but physically or socially nonsensical.

Why Physical and Social Intuition Matter

Language models trained purely on text can excel at linguistic patterns without truly understanding how the world works. HellaSwag exposes this gap by requiring models to reason about:

  • Physical constraints: "After mixing the batter..." shouldn't continue with "...the bowl floats to the ceiling"
  • Social norms: "She waved goodbye, then..." shouldn't end with "...punched the friendly neighbor"
  • Temporal coherence: Events must follow realistic sequences

Think of it like testing if someone has actually lived in the world versus just read about it. A model might assign high probability to grammatically correct text that describes impossible physics.

How It Differs from Perplexity

Perplexity measures average surprise across all tokens. HellaSwag requires discriminative reasoning—choosing the most sensible continuation among deceptive alternatives. A model with decent perplexity might still fail if it can't distinguish common sense from adversarial nonsense.

Key Takeaway: HellaSwag evaluates commonsense reasoning by testing whether models can complete everyday scenarios with physical and social realism, exposing gaps in world understanding that perplexity alone cannot reveal.