Course contentsShow
← Machine Learning and Deep Learning
Lesson 1081 of 3,53824. The Transformer ArchitectureFree lesson

Sinusoidal Positional Encoding: Mathematical Formulation

Deriving the sine and cosine functions used in the original Transformer's positional encoding scheme.

Sinusoidal Positional Encoding: Mathematical Formulation

You'll learn the exact sine and cosine formulas that give Transformers their sense of position.

The Core Idea

Since Transformers process all tokens in parallel (unlike RNNs), they need an explicit way to inject position information. The original Transformer paper introduced a clever mathematical approach: add positional encodings to embeddings using sine and cosine functions with different frequencies.

The Formula

For a token at position pos and embedding dimension index i, the positional encoding is:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Where:

  • pos = position in the sequence (0, 1, 2, ...)
  • i = dimension index (0 to d_model/2)
  • d_model = total embedding dimension (e.g., 512)

Even-indexed dimensions get sine, odd-indexed dimensions get cosine.

Why This Works

Think of each dimension pair as a clock running at a different speed. Low dimensions (small i) cycle rapidly; high dimensions (large i) cycle slowly. This creates a unique "fingerprint" for each position that the model can learn to decode.

The division by 10000^(2i/d_model) creates wavelengths ranging from 2π (fast oscillations) to 10000·2π (slow oscillations). Different positions maintain consistent relative distances across these frequencies, helping the model learn relationships like "3 positions apart."

Key Takeaway: Sinusoidal positional encoding uses alternating sine and cosine functions at exponentially decreasing frequencies to create unique, learnable position signatures for every token position in the sequence.