Course contentsShow
← Machine Learning and Deep Learning
Lesson 3153 of 3,53869. LLM EvaluationFree lesson

BBH: BIG-Bench Hard Tasks

Challenging reasoning tasks from BIG-Bench where previous models struggled significantly.

BBH: BIG-Bench Hard Tasks

What you'll learn: You'll understand what the BIG-Bench Hard (BBH) benchmark is and why it highlights the most challenging reasoning gaps in large language models.

What Is BBH?

BBH is a curated subset of tasks extracted from the larger BIG-Bench collection—specifically, the 23 tasks where even advanced language models performed poorly (often below average human performance). Think of it as a "greatest hits of model failures"—the problems that consistently stumped earlier LLMs and revealed fundamental reasoning weaknesses.

Why These Tasks Matter

While benchmarks like MMLU test breadth of knowledge and HellaSwag tests commonsense reasoning, BBH focuses on complexity and multi-step reasoning. These tasks require models to:

  • Chain multiple logical steps together
  • Understand abstract patterns and relationships
  • Navigate ambiguous or tricky linguistic structures
  • Apply formal reasoning (logical, mathematical, or procedural)

Examples include tasks like navigating boolean expressions, tracking state changes across multiple steps, or understanding subtle semantic distinctions that require careful reasoning rather than pattern matching.

The "Hard" in BBH

The original BIG-Bench contained hundreds of tasks. Researchers identified tasks where the gap between human and model performance was largest, creating BBH as a focused diagnostic tool. If a model performs well on BBH, it suggests genuine reasoning capability rather than memorization or surface-pattern matching.

Key Takeaway: BBH isolates the 23 most difficult reasoning tasks from BIG-Bench where earlier models struggled significantly, serving as a stress test for complex multi-step reasoning abilities that distinguish truly capable language models from those that primarily rely on memorization.