Course contentsShow
← Machine Learning and Deep Learning
Lesson 1108 of 3,53824. The Transformer ArchitectureFree lesson

Long-Range Dependencies Without Gradient Issues

Why transformers handle distant tokens better than RNNs without vanishing/exploding gradients.

Long-Range Dependencies Without Gradient Issues

What you'll learn: How transformers solve the long-range dependency problem that plagued RNNs, without suffering from vanishing or exploding gradients.

The RNN Problem Revisited

Remember how RNNs process sequences step-by-step, passing information forward through time? This sequential processing creates a chain of gradient computations during backpropagation. When you need information from token 1 to influence token 100, the gradient must flow backward through 99 steps—and with each step, it can vanish (shrink toward zero) or explode (grow uncontrollably).

Even LSTMs and GRUs, which help with gradient flow through their gating mechanisms, still struggle with very long sequences because they fundamentally process information sequentially.

The Transformer Solution

Transformers handle distant tokens completely differently. Through self-attention, every token directly computes relationships with every other token in a single step. Token 1 and token 100 are just one attention operation apart—no matter how far apart they are in the sequence.

When gradients flow backward during training, they travel through these direct connections. There's no chain of 99 intermediate steps to navigate. The gradient path from any token to any other token is constant length, regardless of their distance in the sequence.

Why This Matters

Think of it like a highway system versus walking through a chain of villages. RNNs make information walk village-by-village (sequential steps), losing energy along the way. Transformers build direct highways between every location (attention connections), preserving information strength regardless of distance.

This architectural difference, combined with the parallelization we discussed earlier, is why transformers have become the dominant architecture for sequence tasks—especially those requiring understanding of long-range relationships.

Key Takeaway: Transformers eliminate gradient vanishing/exploding over long distances by creating direct attention-based connections between all tokens, making the gradient path length constant rather than proportional to sequence distance.