Course contentsShow
← Machine Learning and Deep Learning
Lesson 1087 of 3,53824. The Transformer ArchitectureFree lesson

Relative Positional Encodings in Transformers

Introducing methods that encode relative distances between tokens rather than absolute positions.

Relative Positional Encodings in Transformers

What you'll learn: How Transformers can encode the distance between tokens rather than their absolute positions in a sequence.

The Core Problem with Absolute Positions

You've learned about absolute positional encodings — they tell the model "this token is at position 5" or "position 20." But consider this analogy: when reading a sentence, you often care more about how far apart two words are than their exact positions. "The cat" means the same thing whether it appears at the start or middle of a paragraph.

What Are Relative Positional Encodings?

Relative positional encodings modify the attention mechanism to incorporate the relative distance between tokens. Instead of saying "token at position 3 attends to token at position 7," we say "this token attends to another token 4 positions away."

The key insight: when computing attention scores between query position i and key position j, we add a learnable bias term based on the offset (i - j). This allows the model to learn patterns like "words 2 positions apart often relate this way."

Why This Matters

Relative encodings offer several advantages:

  • Better generalization: The model learns relationships independent of absolute position
  • Length flexibility: Patterns learned on short sequences transfer to longer ones
  • More natural: Captures how language actually works (local dependencies matter more than absolute location)

Instead of fixed sine/cosine waves or learned absolute embeddings, relative methods focus the model's attention on relationships rather than locations.

Key Takeaway: Relative positional encodings teach Transformers to care about "how far apart" tokens are rather than "where they are," leading to better generalization and more linguistically natural representations.