Course contentsShow
← Machine Learning and Deep Learning
Lesson 913 of 3,53821. CNN ArchitecturesFree lesson

Residual Networks in Practice

Implementation tips: initialization, learning rates, and training strategies for ResNets.

Residual Networks in Practice

You'll learn: How to properly initialize, configure learning rates, and train ResNets to maximize performance and stability.

Core Idea

Building ResNets isn't just about stacking residual blocks—getting them to train efficiently requires careful attention to initialization, learning rate schedules, and training strategies that account for their unique skip connection architecture.

Initialization Tips

He Initialization (also called Kaiming initialization) works best for ResNets with ReLU activations. It accounts for the fact that each layer has roughly the same variance in activations, which is crucial when gradients flow through many layers.

For pre-activation ResNets, initialize batch normalization layers with small weights (often γ=0 for the final BN in each block). This makes the network behave like a shallower network early in training, improving stability.

Learning Rate Strategy

Start with a higher learning rate than you might use for plain networks—ResNets' improved gradient flow allows this. A common approach:

  • Initial LR: 0.1 for batch size 256
  • Warmup: Gradually increase LR over the first few epochs (prevents early instability)
  • Step decay: Reduce by 10× at 30%, 60%, and 90% of total epochs
  • Or use cosine annealing for smoother decay

Training Strategies

  • Longer training: ResNets benefit from extended training (180-200 epochs on ImageNet)
  • Weight decay: Use moderate regularization (1e-4) to prevent overfitting
  • Data augmentation: Standard crops, flips, and color jittering work well
  • Batch size: Larger batches (256+) are common; scale LR proportionally

Think of it like breaking in a new car—you need a careful warm-up period, then you can push harder, but consistent maintenance (regularization) keeps it running smoothly.

Key Takeaway: ResNets train best with He initialization, learning rate warmup followed by aggressive schedules, and patient training with strong regularization—their skip connections enable these aggressive strategies that would destabilize plain networks.