MMLU: Massive Multitask Language Understanding
You'll learn what MMLU measures and why it's become a standard for evaluating LLM world knowledge across diverse domains.
What Is MMLU?
MMLU is a comprehensive benchmark consisting of 57 subjects spanning elementary mathematics to professional law, history, computer science, medicine, and more. Each subject contains multiple-choice questions designed to test an LLM's accumulated knowledge and problem-solving ability.
Think of MMLU as a "universal academic exam" for language models—like testing whether a student can ace freshman biology and graduate-level physics and professional ethics simultaneously.
Why MMLU Matters
Unlike perplexity (which measures language modeling quality), MMLU directly evaluates whether models possess factual knowledge and can apply reasoning across domains. It answers: "Does this model understand the world broadly, or just generate plausible-sounding text?"
The benchmark includes:
- STEM subjects: abstract algebra, college chemistry, electrical engineering
- Humanities: world religions, moral scenarios, philosophy
- Social sciences: sociology, US government, jurisprudence
- Other: professional medicine, nutrition, marketing
Models are tested in a few-shot setting—given a handful of example questions before answering new ones—mimicking how humans learn task formats quickly.
Interpreting MMLU Scores
Scores are reported as accuracy percentages per subject and overall. Random guessing on 4-choice questions yields 25% baseline. State-of-the-art LLMs now exceed 85%, but performance varies dramatically by subject—revealing knowledge gaps in specialized domains like college mathematics or virology.
MMLU's strength is its breadth: a model scoring well must demonstrate genuine multitask understanding, not just memorize narrow training data patterns.
Key Takeaway: MMLU evaluates LLM world knowledge and reasoning across 57 diverse subjects through multiple-choice questions, serving as a standardized "academic proficiency test" that reveals both breadth and domain-specific strengths or weaknesses.