Sinusoidal Positional Encoding: Mathematical Formulation
You'll learn the exact sine and cosine formulas that give Transformers their sense of position.
The Core Idea
Since Transformers process all tokens in parallel (unlike RNNs), they need an explicit way to inject position information. The original Transformer paper introduced a clever mathematical approach: add positional encodings to embeddings using sine and cosine functions with different frequencies.
The Formula
For a token at position pos and embedding dimension index i, the positional encoding is:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
Where:
pos= position in the sequence (0, 1, 2, ...)i= dimension index (0 to d_model/2)d_model= total embedding dimension (e.g., 512)
Even-indexed dimensions get sine, odd-indexed dimensions get cosine.
Why This Works
Think of each dimension pair as a clock running at a different speed. Low dimensions (small i) cycle rapidly; high dimensions (large i) cycle slowly. This creates a unique "fingerprint" for each position that the model can learn to decode.
The division by 10000^(2i/d_model) creates wavelengths ranging from 2π (fast oscillations) to 10000·2π (slow oscillations). Different positions maintain consistent relative distances across these frequencies, helping the model learn relationships like "3 positions apart."
Key Takeaway: Sinusoidal positional encoding uses alternating sine and cosine functions at exponentially decreasing frequencies to create unique, learnable position signatures for every token position in the sequence.