Course contentsShow
← Machine Learning and Deep Learning
Lesson 1608 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Residual Connections in Deep Transformers

Understand how residual connections enable gradient flow through dozens or hundreds of transformer layers.

Residual Connections in Deep Transformers

What you'll learn: How residual connections allow transformers to scale to hundreds of layers without losing gradient signal during training.

The Vanishing Gradient Problem in Deep Networks

When you stack dozens or hundreds of transformer layers, gradients during backpropagation must flow backward through all those layers. Without special architecture tricks, these gradients shrink exponentially—a problem called vanishing gradients. By the time the gradient reaches early layers, it's practically zero, so those layers can't learn.

How Residual Connections Solve This

A residual connection (or skip connection) adds the input of a layer directly to its output:

output = LayerOperation(input) + input

Instead of forcing each layer to learn the complete transformation, the layer only learns the difference (residual) from the identity mapping. The gradient can now flow directly backward through the addition operation, bypassing the layer's complex computations.

In Transformer Layers

Each transformer layer typically has two residual connections:

  1. Around self-attention: x = x + Attention(Norm(x))
  2. Around feed-forward: x = x + FFN(Norm(x))

This creates "highways" for gradients to travel backward through the entire network. Even in a 100-layer transformer, gradients can flow almost unchanged through these direct paths, while each layer fine-tunes the representation incrementally.

The Analogy

Think of a deep transformer like a relay race where each runner (layer) passes the baton (information). Without residual connections, each runner must throw the baton to the next—it degrades with each throw. With residual connections, runners hand it directly, ensuring it arrives intact at the finish line.

Key Takeaway: Residual connections enable gradient flow through dozens or hundreds of transformer layers by providing direct pathways that bypass complex computations, making ultra-deep language models like GPT-3 trainable.