Course contentsShow
← Machine Learning and Deep Learning
Lesson 1083 of 3,53824. The Transformer ArchitectureFree lesson

Implementing Sinusoidal Positional Encoding

Writing code to generate sinusoidal position embeddings for sequences of arbitrary length.

Implementing Sinusoidal Positional Encoding

What you'll learn: How to write Python code that generates sinusoidal position embeddings for any sequence length.

The Core Idea

Now that you understand why sinusoidal functions work mathematically, it's time to turn the formulas into actual code. You'll create a function that takes a sequence length and model dimension, then outputs a matrix where each row represents a position and each column captures positional information using alternating sine and cosine waves at different frequencies.

Building the Positional Encoding Matrix

Think of this as filling in a grid: rows are positions (0, 1, 2, ..., seq_len-1), and columns alternate between sine and cosine applied to those positions at different wavelengths. Here's the implementation:

import torch
import math

def sinusoidal_positional_encoding(seq_len, d_model):
    # Create position indices [0, 1, 2, ..., seq_len-1]
    position = torch.arange(seq_len).unsqueeze(1)  # Shape: (seq_len, 1)
    
    # Create dimension indices [0, 2, 4, ..., d_model-2]
    div_term = torch.exp(torch.arange(0, d_model, 2) * 
                         -(math.log(10000.0) / d_model))
    
    # Initialize positional encoding matrix
    pe = torch.zeros(seq_len, d_model)
    
    # Apply sin to even indices
    pe[:, 0::2] = torch.sin(position * div_term)
    
    # Apply cos to odd indices
    pe[:, 1::2] = torch.cos(position * div_term)
    
    return pe

You can now call sinusoidal_positional_encoding(100, 512) to get position embeddings for sequences up to length 100 with a 512-dimensional model.

Key Takeaway: Implementing sinusoidal positional encoding involves creating a matrix where each position uses sine/cosine functions at progressively slower frequencies across dimensions, giving the Transformer unique, meaningful position information for every token.