RoPE: Mathematical Foundation
What You'll Learn
You'll learn how Rotary Positional Embeddings (RoPE) mathematically encode positions by rotating embedding vectors in a way that preserves relative position information through dot products.
The Core Mathematical Idea
RoPE works by applying rotation matrices to your query and key vectors. Think of it like a clock: if you rotate the hour hand by 30° and the minute hand by 60°, the angle between them changes predictably—that's how RoPE encodes relative position.
Why Rotation Matrices?
A 2D rotation matrix rotates a vector by angle θ:
R(θ) = [cos(θ) -sin(θ)]
[sin(θ) cos(θ)]
The brilliant insight: if you rotate query vector q by angle mθ (where m is its position) and key vector k by angle nθ (position n), then their dot product q·k contains a rotation by (m-n)θ. This captures the relative position m-n, not absolute positions!
The Mathematical Property
For positions m and n:
- Rotate q at position m: R(mθ)q
- Rotate k at position n: R(nθ)k
- Their dot product: (R(mθ)q)ᵀ(R(nθ)k) = qᵀR((m-n)θ)k
This means the attention score depends only on the relative distance (m-n), not where m and n are individually.
Extending to Higher Dimensions
For higher-dimensional embeddings, RoPE applies different rotation frequencies to different dimension pairs, creating a multi-frequency encoding that uniquely identifies each relative position.
Key Takeaway: RoPE uses rotation matrices to embed positions directly into query/key vectors, ensuring that attention scores naturally depend on relative positions through geometric properties of rotations, not learned patterns.