Course contentsShow
← Machine Learning and Deep Learning
Lesson 901 of 3,53821. CNN ArchitecturesFree lesson

The Degradation Problem in Deep Networks

Understanding why simply stacking more layers leads to higher training error, not just overfitting.

The Degradation Problem in Deep Networks

What you'll learn: Why simply adding more layers to a neural network can paradoxically make training worse, even on the training set itself.

The Counterintuitive Discovery

In the early 2010s, researchers made a puzzling observation: when they kept adding layers to deep CNNs, performance would improve up to a certain depth, then start to degrade. Surprisingly, this wasn't just overfitting (poor generalization to test data) — the training error itself got worse with more layers.

Why This Doesn't Make Sense (At First)

Logically, a deeper network should never perform worse than a shallow one. Here's why: imagine you have a 20-layer network that works well. If you add 10 more layers, those new layers could theoretically learn to just copy their input (the identity function), making the 30-layer network functionally identical to the 20-layer one. So deeper should be at least as good, right?

What Actually Happens

In practice, optimization becomes extremely difficult. When networks get very deep:

  • Gradients vanish or explode during backpropagation
  • Optimization algorithms struggle to find good solutions
  • The network learns a suboptimal mapping that's actually worse than shallow networks

This isn't about the model's capacity — it's about our ability to train that capacity effectively. The degradation problem reveals that simply stacking more layers creates training difficulties that standard optimization techniques can't overcome.

Key Takeaway: The degradation problem showed that deeper networks don't automatically mean better performance; very deep networks suffered from higher training error due to optimization difficulties, not just overfitting, motivating the need for new architectural innovations like residual connections.