Computing Perplexity on Test Sets
What you'll learn: How to actually calculate perplexity values from a tokenized test sequence using a trained language model.
The Computation Process
Now that you understand what perplexity measures and how it derives from cross-entropy, let's walk through the mechanical steps to compute it on real test data.
Step-by-Step Calculation
1. Tokenize your test sequence: Convert your text into token IDs that your model understands. If your test set has multiple sequences, you'll typically compute perplexity over each and may report an average.
2. Forward pass through the model: Feed the tokens through your language model to get predicted probability distributions at each position. The model outputs probabilities for the next token given all previous tokens.
3. Extract true token probabilities: For each position i, look up the probability the model assigned to the actual token that appeared in the test sequence: p(token_i | token_1, ..., token_{i-1}).
4. Compute log-likelihood: Sum the log probabilities: log_likelihood = Σ log(p(token_i)) across all N tokens.
5. Calculate perplexity: Use the formula perplexity = exp(-log_likelihood / N). The negative sign converts the typically negative log-likelihood to positive, and dividing by N normalizes by sequence length.
Practical Considerations
- Padding tokens: Exclude padding from your count—only compute over actual content tokens
- Special tokens: Decide whether to include
[CLS],[SEP], or other special markers - Numerical stability: Work in log-space throughout to avoid underflow with tiny probabilities
- Batch processing: Compute perplexity for each sequence individually, then average if reporting dataset-level metrics
Key Takeaway: Computing perplexity requires a forward pass to get token probabilities, extracting the probability of each actual token, computing average negative log-likelihood, and exponentiating the result—carefully handling special tokens and numerical precision throughout.