The Degradation Problem in Deep Networks
What you'll learn: Why simply adding more layers to a neural network can paradoxically make training worse, even on the training set itself.
The Counterintuitive Discovery
In the early 2010s, researchers made a puzzling observation: when they kept adding layers to deep CNNs, performance would improve up to a certain depth, then start to degrade. Surprisingly, this wasn't just overfitting (poor generalization to test data) — the training error itself got worse with more layers.
Why This Doesn't Make Sense (At First)
Logically, a deeper network should never perform worse than a shallow one. Here's why: imagine you have a 20-layer network that works well. If you add 10 more layers, those new layers could theoretically learn to just copy their input (the identity function), making the 30-layer network functionally identical to the 20-layer one. So deeper should be at least as good, right?
What Actually Happens
In practice, optimization becomes extremely difficult. When networks get very deep:
- Gradients vanish or explode during backpropagation
- Optimization algorithms struggle to find good solutions
- The network learns a suboptimal mapping that's actually worse than shallow networks
This isn't about the model's capacity — it's about our ability to train that capacity effectively. The degradation problem reveals that simply stacking more layers creates training difficulties that standard optimization techniques can't overcome.
Key Takeaway: The degradation problem showed that deeper networks don't automatically mean better performance; very deep networks suffered from higher training error due to optimization difficulties, not just overfitting, motivating the need for new architectural innovations like residual connections.