Residual Connections in Deep Transformers
What you'll learn: How residual connections allow transformers to scale to hundreds of layers without losing gradient signal during training.
The Vanishing Gradient Problem in Deep Networks
When you stack dozens or hundreds of transformer layers, gradients during backpropagation must flow backward through all those layers. Without special architecture tricks, these gradients shrink exponentially—a problem called vanishing gradients. By the time the gradient reaches early layers, it's practically zero, so those layers can't learn.
How Residual Connections Solve This
A residual connection (or skip connection) adds the input of a layer directly to its output:
output = LayerOperation(input) + input
Instead of forcing each layer to learn the complete transformation, the layer only learns the difference (residual) from the identity mapping. The gradient can now flow directly backward through the addition operation, bypassing the layer's complex computations.
In Transformer Layers
Each transformer layer typically has two residual connections:
- Around self-attention:
x = x + Attention(Norm(x)) - Around feed-forward:
x = x + FFN(Norm(x))
This creates "highways" for gradients to travel backward through the entire network. Even in a 100-layer transformer, gradients can flow almost unchanged through these direct paths, while each layer fine-tunes the representation incrementally.
The Analogy
Think of a deep transformer like a relay race where each runner (layer) passes the baton (information). Without residual connections, each runner must throw the baton to the next—it degrades with each throw. With residual connections, runners hand it directly, ensuring it arrives intact at the finish line.
Key Takeaway: Residual connections enable gradient flow through dozens or hundreds of transformer layers by providing direct pathways that bypass complex computations, making ultra-deep language models like GPT-3 trainable.