Absolute vs Relative Positional Encoding
What you'll learn: How Transformers can encode position information either as fixed locations in a sequence or as distances between tokens.
The Core Distinction
Since Transformers process all tokens in parallel (unlike RNNs), they need an explicit way to understand word order. There are two fundamental approaches:
Absolute positional encoding assigns each position in a sequence a unique identifier. Think of it like house numbers on a street: the first token is "position 0," the second is "position 1," and so on. The model learns that "position 3" has specific properties, regardless of what other tokens are present.
Relative positional encoding instead captures the distance between tokens. Rather than "I'm at position 3," it says "I'm 2 positions away from that other token." This is like giving directions: "turn left at the second intersection" rather than "go to coordinates (42.3, -71.1)."
Why the Difference Matters
Absolute encoding is simpler and was used in the original Transformer. However, it struggles with sequences longer than those seen during training—the model has never encountered "position 1000" if it only trained on sequences up to length 512.
Relative encoding is more flexible. If the model learns that adjacent words often relate strongly, this pattern holds regardless of whether those words appear at positions 5-6 or 500-501. It generalizes better to different sequence lengths.
Analogy
Imagine giving someone directions to your seat in a theater. Absolute encoding: "I'm in seat D-17." Relative encoding: "I'm three seats to your left." The relative approach works even if you move to a different row—the spatial relationship concept transfers.
Key Takeaway: Absolute positional encoding assigns fixed identifiers to each position in a sequence, while relative encoding captures distances between tokens, offering better generalization to varying sequence lengths.