Course contentsShow
← Machine Learning and Deep Learning
Lesson 1092 of 3,53824. The Transformer ArchitectureFree lesson

Positional Encoding for Long Context

Understanding how different positional encoding schemes handle extrapolation beyond training sequence lengths.

Positional Encoding for Long Context

What you'll learn: How positional encoding methods behave when sequences grow longer than those seen during training.

The Extrapolation Challenge

When you train a Transformer on sequences up to, say, 512 tokens long, what happens when you need to process a 2,048-token document at inference time? This is the extrapolation problem—can your positional encoding gracefully handle positions it's never seen before?

How Different Methods Handle Long Context

Sinusoidal encodings were designed with extrapolation in mind. Their smooth, continuous mathematical functions naturally extend beyond training lengths. However, the model may still struggle because it hasn't learned to attend across such long distances.

Learned positional embeddings face a hard wall—they only have explicit vectors for positions seen during training. Position 1,000 simply doesn't exist if you trained on sequences up to 512 tokens.

RoPE (Rotary Positional Embeddings) generally extrapolates better than absolute methods because it encodes relative distances through rotations. The geometry stays consistent even at unseen positions.

ALiBi (Attention with Linear Biases) shines here—it adds distance-based penalties directly to attention scores. Since it's just arithmetic (not learned parameters), it naturally scales to any sequence length without modification.

Think of it like measuring distance: absolute addresses ("Building 512") don't help beyond your neighborhood, but relative measures ("5 blocks away") work anywhere.

Practical Implications

For long-document applications, methods like ALiBi and RoPE show superior extrapolation. For fixed-length tasks, simpler approaches suffice. This is why modern large language models increasingly adopt RoPE or ALiBi for their long-context capabilities.

Key Takeaway: Positional encoding methods differ dramatically in handling sequences longer than training data—relative and bias-based approaches (RoPE, ALiBi) extrapolate better than absolute learned embeddings, making them crucial for long-context applications.