Course contentsShow
← Machine Learning and Deep Learning
Lesson 1089 of 3,53824. The Transformer ArchitectureFree lesson

RoPE: Mathematical Foundation

Deriving how RoPE uses rotation matrices to encode relative positions while preserving dot product structure.

RoPE: Mathematical Foundation

What You'll Learn

You'll learn how Rotary Positional Embeddings (RoPE) mathematically encode positions by rotating embedding vectors in a way that preserves relative position information through dot products.

The Core Mathematical Idea

RoPE works by applying rotation matrices to your query and key vectors. Think of it like a clock: if you rotate the hour hand by 30° and the minute hand by 60°, the angle between them changes predictably—that's how RoPE encodes relative position.

Why Rotation Matrices?

A 2D rotation matrix rotates a vector by angle θ:

R(θ) = [cos(θ)  -sin(θ)]
       [sin(θ)   cos(θ)]

The brilliant insight: if you rotate query vector q by angle mθ (where m is its position) and key vector k by angle nθ (position n), then their dot product q·k contains a rotation by (m-n)θ. This captures the relative position m-n, not absolute positions!

The Mathematical Property

For positions m and n:

  • Rotate q at position m: R(mθ)q
  • Rotate k at position n: R(nθ)k
  • Their dot product: (R(mθ)q)ᵀ(R(nθ)k) = qᵀR((m-n)θ)k

This means the attention score depends only on the relative distance (m-n), not where m and n are individually.

Extending to Higher Dimensions

For higher-dimensional embeddings, RoPE applies different rotation frequencies to different dimension pairs, creating a multi-frequency encoding that uniquely identifies each relative position.

Key Takeaway: RoPE uses rotation matrices to embed positions directly into query/key vectors, ensuring that attention scores naturally depend on relative positions through geometric properties of rotations, not learned patterns.