Course contentsShow
← Machine Learning and Deep Learning
Lesson 1611 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Rotary Position Embeddings (RoPE)

Understand how RoPE encodes position through rotation in embedding space, enabling length extrapolation.

Rotary Position Embeddings (RoPE)

You'll learn: How modern LLMs encode positional information by rotating embeddings in high-dimensional space, enabling them to handle sequences longer than those seen during training.

The Core Idea

Traditional position embeddings add a fixed vector to each token based on its position. RoPE takes a different approach: it rotates the embedding vectors in pairs of dimensions, where the rotation angle depends on the token's position.

Think of it like a clock: each position gets a unique rotation angle. When computing attention between two tokens, the dot product naturally captures their relative position difference through these rotations—just like you can tell time by comparing clock hands' angles.

How RoPE Works

RoPE applies rotation matrices to query and key vectors during attention:

  1. Split dimensions into pairs: Your embedding vector is treated as multiple 2D planes
  2. Rotate each pair: Apply position-dependent rotation angles (θ₀, θ₁, θ₂...)
  3. Compute attention: The rotated queries and keys naturally encode relative position

The rotation angle for position m in dimension pair i is: m · θᵢ, where θᵢ = 10000^(-2i/d).

Why It Enables Length Extrapolation

Because RoPE encodes relative positions through rotations, the model learns patterns like "attend to the token 5 positions back" rather than "attend to absolute position 237." This relative encoding generalizes better to longer sequences, allowing inference on texts much longer than training examples—a critical capability for modern LLMs.

Key Takeaway: RoPE encodes position by rotating embedding pairs in high-dimensional space, creating relative position encodings that generalize naturally to longer sequences than seen during training.