Course contentsShow
← Machine Learning and Deep Learning
Lesson 748 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Stochastic Depth

Randomly dropping entire layers during training to regularize very deep networks like ResNets.

Stochastic Depth

What You'll Learn

You'll discover how Stochastic Depth randomly drops entire layers during training to prevent overfitting in very deep networks like ResNets.

The Core Idea

Imagine you're training a 100-layer neural network. Instead of always using all 100 layers during each training step, Stochastic Depth randomly skips some layers entirely—allowing the gradient to flow through a "shortcut" instead. This is like Dropout, but instead of dropping individual neurons, you're dropping whole layers.

During each forward pass in training:

  • Each layer has a survival probability (often starting high and decreasing with depth)
  • If a layer "survives," data flows through it normally
  • If it "dies," the input skips directly to the next layer (using residual connections)

At inference time, all layers are active, but their outputs are scaled by their survival probability—just like inverted dropout.

Why This Works

Stochastic Depth acts as powerful regularization for very deep networks:

  • Reduces co-adaptation: The network can't rely on any single layer always being present
  • Implicit ensemble: You're training many sub-networks of varying depths simultaneously
  • Better gradient flow: Shorter paths during training help gradients reach early layers
  • Prevents overfitting: Especially useful in networks with 50+ layers

This technique is particularly effective with ResNets and other architectures that have skip connections, since those shortcuts provide the "highway" when layers are dropped.

Analogy

Think of a relay race where runners occasionally take shortcuts. The team learns to be flexible—no single runner is critical, and the team becomes more robust overall.

Key Takeaway: Stochastic Depth randomly drops entire layers during training to regularize very deep networks, creating an ensemble of sub-networks while improving gradient flow.