Course contentsShow
← Machine Learning and Deep Learning
Lesson 1079 of 3,53824. The Transformer ArchitectureFree lesson

Why Transformers Need Positional Information

Understanding why self-attention is permutation-invariant and requires explicit position encoding to process sequences.

Why Transformers Need Positional Information

What you'll learn: You'll understand why the Transformer architecture can't process word order naturally and needs explicit help to know which word comes first, second, third, and so on.

The Permutation Problem

Remember how self-attention computes a weighted sum of values based on query-key similarity? Here's the catch: that operation doesn't care about order at all.

If you scramble the words in a sentence—say, change "The cat sat" to "sat The cat"—the attention mechanism will produce exactly the same outputs, just in shuffled positions. It's like giving someone a bag of numbered puzzle pieces but removing all the numbers: they can see the pieces but can't assemble them in order.

Why This Happens

Self-attention treats the input sequence like a set rather than a sequence. When computing attention scores between queries and keys, the dot products only depend on the content of the vectors, not their positions. The softmax normalization and weighted sum operations are also position-agnostic.

A Simple Analogy

Imagine reading words printed on separate cards scattered on a table. You can see all the words simultaneously and understand relationships between them, but unless the cards are numbered or arranged in a line, you can't tell if the sentence is "Dog bites man" or "Man bites dog"—very different meanings!

The Solution Preview

Since transformers process all positions simultaneously (unlike RNNs which step through sequentially), they need us to explicitly inject position information into the input embeddings. Without this, the model is blind to word order.

Key Takeaway: Self-attention is permutation-invariant by design—it treats sequences as unordered sets—so Transformers require explicit positional encoding to understand that sequence order matters.