Course contentsShow
← Machine Learning and Deep Learning
Lesson 1082 of 3,53824. The Transformer ArchitectureFree lesson

Why Sinusoidal Functions Work for Position

Understanding the wavelength properties and linearity that make sinusoidal encodings effective for position representation.

Why Sinusoidal Functions Work for Position

You'll learn why sine and cosine waves create ideal positional encodings for Transformers.

The Core Idea

Sinusoidal functions (sine and cosine) aren't arbitrary choices—they have unique mathematical properties that make them perfect for representing position. Two key properties stand out: wavelength variation and linear extrapolation.

Wavelength Properties

Each dimension in sinusoidal encoding uses a different frequency (wavelength). Lower frequencies create slow-changing patterns that capture coarse position differences (like "early in sequence" vs "late in sequence"). Higher frequencies oscillate rapidly, capturing fine-grained distinctions between adjacent positions.

Think of it like musical octaves: bass notes distinguish broad sections of a song, while high notes differentiate individual beats. This multi-scale representation lets the model recognize both local and global position relationships simultaneously.

Linear Combinations Enable Relative Position

Here's the mathematical magic: because of trigonometric identities, the encoding at position pos + k can be represented as a linear function of the encoding at position pos. Specifically:

sin(pos + k) = sin(pos)·cos(k) + cos(pos)·sin(k)

This means the model can learn to compute relative position offsets through simple linear transformations. It naturally learns "how far apart are these tokens?" rather than memorizing absolute positions.

Smooth and Continuous

Unlike learned embeddings that are discrete lookup tables, sinusoidal functions are smooth and continuous. This means positions the model has never seen (longer sequences at inference) still receive meaningful encodings through natural extrapolation.

Key Takeaway: Sinusoidal positional encodings work because their varying wavelengths capture multi-scale position information, and their trigonometric properties allow models to learn relative positions through linear operations—making them both mathematically elegant and practically effective.