Course contentsShow
← Machine Learning and Deep Learning
Lesson 903 of 3,53821. CNN ArchitecturesFree lesson

Residual Learning Formulation

Learning F(x) = H(x) - x instead of H(x) directly, and why this is easier to optimize.

Residual Learning Formulation

What you'll learn: How reformulating the learning problem from H(x) to F(x) = H(x) - x makes training deep networks dramatically easier.

The Core Idea

In traditional deep learning, we ask our network to learn a mapping H(x) directly — the complete transformation from input x to desired output. But ResNets flip this approach: instead of learning H(x), they learn the residual F(x) = H(x) - x, which represents the difference between what we want and what we already have.

The final output becomes: H(x) = F(x) + x

The skip connection provides the "x" part automatically, so the network only needs to learn the "correction" F(x).

Why This Is Easier to Optimize

Think of it like giving directions. Instead of saying "walk to the corner of 5th and Main" (learning H(x) from scratch), you say "walk two blocks forward from where you are" (learning the residual F(x)). The second approach builds on your current position.

For neural networks:

  • Identity mapping is trivial: If the optimal transformation is close to identity (output ≈ input), the network just needs to learn F(x) ≈ 0, which is easier than learning H(x) ≈ x
  • Gradient flow: The skip connection provides a direct path for gradients during backpropagation, alleviating the vanishing gradient problem
  • Incremental refinement: Each layer refines the representation slightly rather than reconstructing everything

Instead of building features from scratch at every layer, residual learning lets deeper layers make small adjustments to what earlier layers already computed.

Key Takeaway: Residual learning reformulates the problem as learning F(x) = H(x) - x instead of H(x) directly, making it easier for networks to learn identity mappings and enabling effective training of very deep architectures through better gradient flow and incremental feature refinement.