Course contentsShow
← Machine Learning and Deep Learning
Lesson 900 of 3,53821. CNN ArchitecturesFree lesson

Architectural Evolution: From AlexNet to ResNet

Understand the progression from deeper networks to the need for skip connections and residual learning.

Architectural Evolution: From AlexNet to ResNet

What you'll learn: How the push for deeper networks revealed critical training challenges that led to the invention of residual connections.

The Problem with Going Deeper

After AlexNet's breakthrough in 2012, researchers naturally tried making networks deeper—more layers should mean more capacity to learn complex patterns, right? VGGNet pushed to 16-19 layers, and it worked. But something strange happened when networks went even deeper: performance actually got worse, even on training data.

This wasn't overfitting (which would show good training performance but poor validation). Networks with 50+ layers were struggling to train at all, performing worse than their shallower counterparts.

The Degradation Problem

The culprit was the degradation problem: as gradients backpropagated through dozens of layers, they became vanishingly small (or exploded). Even with BatchNorm and careful initialization, very deep networks couldn't learn as effectively as medium-depth ones.

Think of it like a game of telephone with 50 people versus 10 people—the message gets increasingly distorted with more participants.

Why Skip Connections Were Revolutionary

ResNet (2015) solved this by introducing skip connections (or residual connections). Instead of forcing each layer to learn the complete transformation, layers now learned just the residual or difference needed. A direct path allowed gradients to flow backward through the network without diminishing.

The insight: if deeper is better, at minimum a deep network should perform as well as a shallow one by just learning to copy the input through extra layers (identity mapping). Skip connections made this trivial.

Key Takeaway: The evolution from AlexNet to ResNet revealed that simply stacking more layers fails due to optimization difficulties; residual learning with skip connections solved the degradation problem, enabling networks with hundreds of layers to train effectively.