Conditional Perplexity and Context Length
What you'll learn: How to measure perplexity with different context window sizes to evaluate how well language models capture long-range dependencies.
The Core Idea
Standard perplexity measures how surprised a model is by text overall, but conditional perplexity examines how well a model predicts tokens given different amounts of preceding context. By varying the context window length, you can assess whether your model truly leverages long-range dependencies or only relies on immediate neighbors.
Think of it like testing reading comprehension: asking someone to predict the next word after seeing just the previous sentence versus after reading several paragraphs. A model that only "understands" local patterns will show similar perplexity regardless of context length, while one that captures long-range structure should improve (lower perplexity) with more context.
Why Context Length Matters
When evaluating language models—especially transformers with large context windows—you want to know:
- Does providing 512 tokens of context actually help versus 128?
- At what context length do diminishing returns appear?
- Are long-range dependencies being modeled or ignored?
Measuring Conditional Perplexity
You compute perplexity on the same test set multiple times, each time using different context window sizes (e.g., 32, 128, 512, 2048 tokens). For each configuration, you calculate how well the model predicts upcoming tokens given that specific amount of history.
A model with effective long-range modeling shows decreasing perplexity as context length increases, plateauing when additional context stops helping. A model that ignores distant context shows flat perplexity curves.
Key Takeaway: Conditional perplexity across varying context lengths reveals whether your language model genuinely exploits long-range dependencies or merely appears competent through local pattern matching.