Relative Positional Encodings in Transformers
What you'll learn: How Transformers can encode the distance between tokens rather than their absolute positions in a sequence.
The Core Problem with Absolute Positions
You've learned about absolute positional encodings — they tell the model "this token is at position 5" or "position 20." But consider this analogy: when reading a sentence, you often care more about how far apart two words are than their exact positions. "The cat" means the same thing whether it appears at the start or middle of a paragraph.
What Are Relative Positional Encodings?
Relative positional encodings modify the attention mechanism to incorporate the relative distance between tokens. Instead of saying "token at position 3 attends to token at position 7," we say "this token attends to another token 4 positions away."
The key insight: when computing attention scores between query position i and key position j, we add a learnable bias term based on the offset (i - j). This allows the model to learn patterns like "words 2 positions apart often relate this way."
Why This Matters
Relative encodings offer several advantages:
- Better generalization: The model learns relationships independent of absolute position
- Length flexibility: Patterns learned on short sequences transfer to longer ones
- More natural: Captures how language actually works (local dependencies matter more than absolute location)
Instead of fixed sine/cosine waves or learned absolute embeddings, relative methods focus the model's attention on relationships rather than locations.
Key Takeaway: Relative positional encodings teach Transformers to care about "how far apart" tokens are rather than "where they are," leading to better generalization and more linguistically natural representations.