Course contentsShow
← Machine Learning and Deep Learning
Lesson 902 of 3,53821. CNN ArchitecturesFree lesson

Skip Connections: The Core Idea

How identity mappings allow gradients to flow directly backward through the network.

Skip Connections: The Core Idea

What you'll learn: How identity mappings in skip connections solve the gradient flow problem that prevents very deep networks from training effectively.

The Problem Skip Connections Solve

In the previous lesson, you learned that deep networks suffer from the degradation problem—adding more layers actually makes performance worse, not from overfitting but from optimization difficulty. The core issue? Gradients get weaker as they backpropagate through many layers, making it hard for early layers to learn anything useful.

How Skip Connections Work

Skip connections (or residual connections) provide a shortcut path for gradients to flow backward through the network. Instead of forcing information to pass through every transformation, you create an alternative route that bypasses one or more layers.

Think of it like a highway system: the main road goes through every town (the layers), but you also build an express lane that goes straight through. Traffic (gradients) can flow on both paths simultaneously.

The Identity Mapping

The "skip" is typically an identity mapping—the input passes through unchanged and gets added to the output of the skipped layers. Mathematically, instead of learning F(x), the network learns F(x) + x.

During backpropagation, gradients flow through both the computational path F(x) AND directly through the identity connection. Even if F(x) has vanishing gradients, the identity path ensures at least some gradient reaches earlier layers. This direct path acts like a gradient highway, preventing the signal from dying out.

Why This Matters

Skip connections don't just help gradients flow—they fundamentally change what the network learns. Layers can now learn small refinements (residuals) rather than complete transformations, making optimization much easier.

Key Takeaway: Skip connections use identity mappings to create direct gradient pathways through the network, ensuring that training signals can flow backward even through very deep architectures.