Constant Path Length Between Tokens
What You'll Learn
You'll understand why self-attention's ability to connect any two tokens in a single step gives Transformers a massive advantage over RNNs for learning long-range dependencies.
The Sequential Distance Problem in RNNs
In an RNN, information flows sequentially from one token to the next. If you want the first word in a sentence to influence the last word, the signal must pass through every intermediate hidden state. For a sentence of length n, this creates a path of length O(n) — the information travels through n steps.
The problem: As the path grows longer, gradients can vanish or explode, and the model struggles to maintain context over long distances. Even with LSTMs and GRUs helping gradient flow, the fundamental sequential bottleneck remains.
Self-Attention's Direct Connections
Self-attention fundamentally changes this architecture. When computing attention, every token directly attends to every other token in a single operation. The first word can directly influence the last word without passing through intermediate steps.
Path length: O(1) — constant, regardless of sequence length.
The Analogy
Think of RNNs like a game of telephone: a message passed person-by-person down a line gets distorted. Self-attention is like everyone in a room speaking directly to everyone else simultaneously — no information loss from relay.
Why This Matters
This constant path length is why Transformers excel at capturing long-range dependencies. Whether tokens are 5 positions apart or 500, the connection strength depends only on their relevance (attention weights), not on how many hops separate them.
Key Takeaway: Self-attention provides O(1) path length between any two tokens, versus RNNs' O(n) sequential distance, enabling direct modeling of long-range dependencies without gradient degradation.