Learned Positional Embeddings
What you'll learn: How to let the model discover the best way to encode position information through training, rather than using fixed sinusoidal functions.
The Core Idea
Instead of using the fixed sinusoidal positional encoding formula we just learned, we can treat positional information like any other embedding—let the model learn it from data. Just as we learn word embeddings during training, we can initialize a lookup table of position embeddings and update them via backpropagation.
Think of it this way: sinusoidal encoding is like giving someone a pre-written dictionary, while learned embeddings are like letting them write their own dictionary based on what works best for their specific task.
How It Works
You create an embedding matrix with shape (max_sequence_length, d_model), where each row represents one position (0, 1, 2, ..., up to your maximum sequence length). These start with random values and get optimized during training.
For a token at position i, you simply look up row i and add it to the token embedding—exactly like you did with sinusoidal encodings, but the values come from a trainable parameter instead of a mathematical formula.
Trade-offs
Pros:
- Can adapt to your specific task and data
- Often performs slightly better in practice
- Simpler to implement
Cons:
- Cannot handle sequences longer than
max_sequence_lengthseen during training - Requires more parameters
- Less mathematically elegant
Many modern Transformers (like BERT and GPT) use learned positional embeddings because they're flexible and effective.
Key Takeaway: Learned positional embeddings replace fixed sinusoidal encodings with trainable parameters, allowing the model to discover optimal position representations for your specific task, though at the cost of flexibility with sequence length.