Course contentsShow
← Machine Learning and Deep Learning
Lesson 3151 of 3,53869. LLM EvaluationFree lesson

HumanEval and Code Generation

Evaluating programming ability through function completion tasks with unit test verification.

HumanEval and Code Generation

What You'll Learn

You'll discover how the HumanEval benchmark evaluates a language model's ability to generate correct Python code by completing functions and verifying them against unit tests.


The Core Idea

HumanEval is a benchmark dataset that measures how well LLMs can write functional code, not just syntactically correct text. Released by OpenAI, it contains 164 hand-written programming problems, each consisting of:

  • A function signature (name and parameters)
  • A docstring describing what the function should do
  • A set of unit tests (hidden from the model) that verify correctness

The model receives only the signature and docstring, then must generate the function body. Success means passing all unit tests for that problem.

Why This Matters

Unlike perplexity or text generation metrics (concepts you've learned), HumanEval tests functional correctness—the code must actually work. This is binary: either it passes all tests or it doesn't. A "pass@k" metric reports the probability that at least one of k generated attempts succeeds, accounting for the stochastic nature of sampling.

Think of it like this: Asking an LLM to complete def is_palindrome(s): with proper docstrings is easy to evaluate—just run test cases like assert is_palindrome("racecar") == True. HumanEval systematizes this across diverse programming challenges (string manipulation, algorithms, data structures).

Evaluation Process

  1. Model generates function completion(s)
  2. Code is inserted into test harness
  3. Unit tests execute in isolated environment
  4. Binary outcome: pass or fail
  5. Aggregate across all 164 problems for overall score

This approach directly measures programming competency rather than proxying through likelihood metrics like perplexity.


Key Takeaway: HumanEval evaluates LLM code generation through function completion tasks verified by comprehensive unit tests, providing a direct measure of functional programming ability rather than statistical text likelihood.