Course contentsShow
← Machine Learning and Deep Learning
Lesson 3137 of 3,53869. LLM EvaluationFree lesson

What Perplexity Measures in Language Models

Understanding perplexity as a measure of how well a model predicts the next token in a sequence.

What Perplexity Measures in Language Models

What you'll learn: How perplexity quantifies a language model's uncertainty when predicting the next token in a sequence.

The Core Idea

Perplexity measures how "surprised" or "confused" a language model is when predicting the next word (or token) in a sequence. A lower perplexity means the model is more confident and accurate in its predictions, while higher perplexity indicates greater uncertainty.

Think of perplexity as measuring how many equally-likely choices the model thinks it has at each step. If a model has perplexity of 10, it's as uncertain as if it were choosing randomly among 10 equally probable words.

The Mathematics Behind It

Perplexity is the exponential of the average cross-entropy loss across all predictions:

Perplexity = exp(average cross-entropy loss)

For a sequence of N tokens, perplexity tells you: "On average, at each position, the model is as confused as if it had to pick from this many equally-likely options."

Why It Matters for LLM Evaluation

Perplexity serves as a go-to intrinsic metric for language models because:

  • It's directly tied to prediction quality: Lower perplexity means the model assigns higher probability to the actual next tokens
  • It's comparable across models: You can use perplexity to compare different architectures on the same test set
  • It captures uncertainty: Unlike accuracy, it penalizes confident wrong predictions more heavily

A model with perplexity of 20 is twice as uncertain as one with perplexity of 10—it behaves as if choosing from twice as many options.

Key Takeaway: Perplexity quantifies a language model's prediction uncertainty by measuring how many equally-probable alternatives it effectively considers at each step—lower is better, indicating more confident and accurate next-token predictions.