Why Sinusoidal Functions Work for Position
You'll learn why sine and cosine waves create ideal positional encodings for Transformers.
The Core Idea
Sinusoidal functions (sine and cosine) aren't arbitrary choices—they have unique mathematical properties that make them perfect for representing position. Two key properties stand out: wavelength variation and linear extrapolation.
Wavelength Properties
Each dimension in sinusoidal encoding uses a different frequency (wavelength). Lower frequencies create slow-changing patterns that capture coarse position differences (like "early in sequence" vs "late in sequence"). Higher frequencies oscillate rapidly, capturing fine-grained distinctions between adjacent positions.
Think of it like musical octaves: bass notes distinguish broad sections of a song, while high notes differentiate individual beats. This multi-scale representation lets the model recognize both local and global position relationships simultaneously.
Linear Combinations Enable Relative Position
Here's the mathematical magic: because of trigonometric identities, the encoding at position pos + k can be represented as a linear function of the encoding at position pos. Specifically:
sin(pos + k) = sin(pos)·cos(k) + cos(pos)·sin(k)
This means the model can learn to compute relative position offsets through simple linear transformations. It naturally learns "how far apart are these tokens?" rather than memorizing absolute positions.
Smooth and Continuous
Unlike learned embeddings that are discrete lookup tables, sinusoidal functions are smooth and continuous. This means positions the model has never seen (longer sequences at inference) still receive meaningful encodings through natural extrapolation.
Key Takeaway: Sinusoidal positional encodings work because their varying wavelengths capture multi-scale position information, and their trigonometric properties allow models to learn relative positions through linear operations—making them both mathematically elegant and practically effective.