Course contentsShow
← Machine Learning and Deep Learning
Lesson 1661 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

YaRN: Yet Another RoPE Scaling

Non-uniform interpolation that preserves local relationships while extending to longer contexts efficiently.

YaRN: Yet Another RoPE Scaling

What You'll Learn

You'll discover how YaRN extends RoPE to longer contexts by scaling different frequency components non-uniformly, preserving important local patterns while enabling greater reach.

The Problem with Uniform Interpolation

When you learned about RoPE Scaling and Interpolation (lesson 1660), you saw how we can extend context windows by interpolating position indices. However, uniform scaling treats all frequency components equally—this can damage the model's ability to capture fine-grained, local relationships that high-frequency components encode.

Non-Uniform Interpolation: The Core Idea

YaRN (Yet another RoPE extensioN) recognizes that different frequency bands in RoPE serve different purposes:

  • High frequencies encode fine-grained, local token relationships (adjacent words, syntax)
  • Low frequencies encode broader, long-range dependencies

YaRN applies different scaling factors to different frequency components. High-frequency components get minimal or no interpolation (preserving local context), while low-frequency components receive more aggressive scaling (extending global reach).

Think of it like adjusting a graphic equalizer: you boost bass (low frequencies) to extend range while keeping treble (high frequencies) clear for local detail.

How It Works

YaRN introduces a frequency-dependent interpolation factor. Instead of scaling all position embeddings uniformly by s, it computes per-dimension scaling based on which frequency band each dimension belongs to. Dimensions corresponding to higher frequencies stay closer to their original values, while lower frequencies get interpolated more aggressively.

This approach maintains the model's ability to understand immediate context (word relationships, grammar) while allowing it to attend much farther back in the sequence.

Key Takeaway: YaRN enables efficient long-context extension by scaling RoPE's frequency components non-uniformly—preserving critical local relationships in high frequencies while stretching low frequencies to reach distant tokens.