Implementing Sinusoidal Positional Encoding
What you'll learn: How to write Python code that generates sinusoidal position embeddings for any sequence length.
The Core Idea
Now that you understand why sinusoidal functions work mathematically, it's time to turn the formulas into actual code. You'll create a function that takes a sequence length and model dimension, then outputs a matrix where each row represents a position and each column captures positional information using alternating sine and cosine waves at different frequencies.
Building the Positional Encoding Matrix
Think of this as filling in a grid: rows are positions (0, 1, 2, ..., seq_len-1), and columns alternate between sine and cosine applied to those positions at different wavelengths. Here's the implementation:
import torch
import math
def sinusoidal_positional_encoding(seq_len, d_model):
# Create position indices [0, 1, 2, ..., seq_len-1]
position = torch.arange(seq_len).unsqueeze(1) # Shape: (seq_len, 1)
# Create dimension indices [0, 2, 4, ..., d_model-2]
div_term = torch.exp(torch.arange(0, d_model, 2) *
-(math.log(10000.0) / d_model))
# Initialize positional encoding matrix
pe = torch.zeros(seq_len, d_model)
# Apply sin to even indices
pe[:, 0::2] = torch.sin(position * div_term)
# Apply cos to odd indices
pe[:, 1::2] = torch.cos(position * div_term)
return pe
You can now call sinusoidal_positional_encoding(100, 512) to get position embeddings for sequences up to length 100 with a 512-dimensional model.
Key Takeaway: Implementing sinusoidal positional encoding involves creating a matrix where each position uses sine/cosine functions at progressively slower frequencies across dimensions, giving the Transformer unique, meaningful position information for every token.