Course contentsShow
← Machine Learning and Deep Learning
Lesson 1662 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Context Length Extrapolation Evaluation

Measuring how well models generalize to sequences longer than seen during training using perplexity and task performance.

Context Length Extrapolation Evaluation

What you'll learn: How to measure whether a language model can handle sequences longer than it saw during training.

The Challenge

Imagine you learned to read sentences up to 10 words long, then suddenly faced a 50-word sentence. Could you still understand it? This is context length extrapolation — the ability of a model to generalize beyond its training context window.

After applying techniques like RoPE scaling, ALiBi, or YaRN to extend context length, you need rigorous evaluation to know if they actually work.

Core Evaluation Metrics

Perplexity on Long Sequences

Perplexity measures how "surprised" the model is by text. Lower is better.

The standard approach:

  1. Take your model trained on, say, 2048-token contexts
  2. Test it on sequences of 4096, 8192, or 16384 tokens
  3. Measure perplexity at each length

If perplexity stays low (or increases slowly), the model extrapolates well. If it spikes dramatically, the model is "confused" by the longer context.

Task Performance

Perplexity alone doesn't tell the whole story. You also evaluate on tasks requiring long-range reasoning:

  • Document QA: Can the model answer questions about information thousands of tokens apart?
  • Passkey Retrieval: Hide a random "passkey" deep in a long document — can the model find it?
  • Multi-Document Tasks: Summarization or analysis spanning multiple full articles

Why Both Matter

Think of perplexity as a health check — it shows the model isn't breaking. Task performance proves it's actually using the extended context productively. A model might maintain decent perplexity but fail to retrieve information from early in a 10K-token sequence, revealing attention degradation.

Key Takeaway: Context length extrapolation evaluation uses perplexity curves across lengths plus targeted long-context tasks to verify that position encoding modifications actually enable models to understand sequences beyond their training distribution.