Course contentsShow
← Machine Learning and Deep Learning
Lesson 1109 of 3,53824. The Transformer ArchitectureFree lesson

Constant Path Length Between Tokens

Understanding how self-attention provides O(1) path length versus RNN's O(n) sequential distance.

Constant Path Length Between Tokens

What You'll Learn

You'll understand why self-attention's ability to connect any two tokens in a single step gives Transformers a massive advantage over RNNs for learning long-range dependencies.

The Sequential Distance Problem in RNNs

In an RNN, information flows sequentially from one token to the next. If you want the first word in a sentence to influence the last word, the signal must pass through every intermediate hidden state. For a sentence of length n, this creates a path of length O(n) — the information travels through n steps.

The problem: As the path grows longer, gradients can vanish or explode, and the model struggles to maintain context over long distances. Even with LSTMs and GRUs helping gradient flow, the fundamental sequential bottleneck remains.

Self-Attention's Direct Connections

Self-attention fundamentally changes this architecture. When computing attention, every token directly attends to every other token in a single operation. The first word can directly influence the last word without passing through intermediate steps.

Path length: O(1) — constant, regardless of sequence length.

The Analogy

Think of RNNs like a game of telephone: a message passed person-by-person down a line gets distorted. Self-attention is like everyone in a room speaking directly to everyone else simultaneously — no information loss from relay.

Why This Matters

This constant path length is why Transformers excel at capturing long-range dependencies. Whether tokens are 5 positions apart or 500, the connection strength depends only on their relevance (attention weights), not on how many hops separate them.

Key Takeaway: Self-attention provides O(1) path length between any two tokens, versus RNNs' O(n) sequential distance, enabling direct modeling of long-range dependencies without gradient degradation.