Residual Connections for Gradient Flow
What You'll Learn
You'll discover how residual (skip) connections create "shortcut highways" for gradients, solving the vanishing gradient problem in very deep networks.
The Highway Analogy
Imagine trying to pass a message through 100 people standing in a line—each person whispers to the next. By the time it reaches the end, the message is garbled or silent. Now imagine you can also shout directly from the start to the end. That's what residual connections do for gradients.
The Core Problem
In deep networks (50+ layers), gradients must flow backward through many layers during backpropagation. Each layer multiplies the gradient by values often less than 1, causing it to shrink exponentially—the vanishing gradient problem we explored earlier. By layer 1, the gradient may be effectively zero, preventing learning.
How Residual Connections Work
A residual connection (or skip connection) adds the input of a layer directly to its output:
output = layer(input) + input
This creates two gradient paths:
- Through the layers: The normal backpropagation path
- Around the layers: A direct shortcut that bypasses transformations
During backpropagation, gradients flow through both paths. The skip connection acts like an express lane—gradients can travel directly backward through the network without degradation, even if the main path suffers from vanishing gradients.
Why It Works
The addition operation in backpropagation distributes gradients equally to both paths. Even if the gradient through the learned layers vanishes, the skip connection preserves a clean gradient signal all the way back to early layers. This enables training networks with 100+ layers (ResNets, for example).
Key Takeaway: Residual connections solve vanishing gradients by providing direct gradient highways that bypass potentially problematic layer transformations, enabling successful training of very deep networks.