Course contentsShow
← Machine Learning and Deep Learning
Lesson 1085 of 3,53824. The Transformer ArchitectureFree lesson

Learned Positional Embeddings

Understanding trainable position embeddings as an alternative to fixed sinusoidal encodings.

Learned Positional Embeddings

What you'll learn: How to let the model discover the best way to encode position information through training, rather than using fixed sinusoidal functions.

The Core Idea

Instead of using the fixed sinusoidal positional encoding formula we just learned, we can treat positional information like any other embedding—let the model learn it from data. Just as we learn word embeddings during training, we can initialize a lookup table of position embeddings and update them via backpropagation.

Think of it this way: sinusoidal encoding is like giving someone a pre-written dictionary, while learned embeddings are like letting them write their own dictionary based on what works best for their specific task.

How It Works

You create an embedding matrix with shape (max_sequence_length, d_model), where each row represents one position (0, 1, 2, ..., up to your maximum sequence length). These start with random values and get optimized during training.

For a token at position i, you simply look up row i and add it to the token embedding—exactly like you did with sinusoidal encodings, but the values come from a trainable parameter instead of a mathematical formula.

Trade-offs

Pros:

  • Can adapt to your specific task and data
  • Often performs slightly better in practice
  • Simpler to implement

Cons:

  • Cannot handle sequences longer than max_sequence_length seen during training
  • Requires more parameters
  • Less mathematically elegant

Many modern Transformers (like BERT and GPT) use learned positional embeddings because they're flexible and effective.

Key Takeaway: Learned positional embeddings replace fixed sinusoidal encodings with trainable parameters, allowing the model to discover optimal position representations for your specific task, though at the cost of flexibility with sequence length.