Residual Networks in Practice
You'll learn: How to properly initialize, configure learning rates, and train ResNets to maximize performance and stability.
Core Idea
Building ResNets isn't just about stacking residual blocks—getting them to train efficiently requires careful attention to initialization, learning rate schedules, and training strategies that account for their unique skip connection architecture.
Initialization Tips
He Initialization (also called Kaiming initialization) works best for ResNets with ReLU activations. It accounts for the fact that each layer has roughly the same variance in activations, which is crucial when gradients flow through many layers.
For pre-activation ResNets, initialize batch normalization layers with small weights (often γ=0 for the final BN in each block). This makes the network behave like a shallower network early in training, improving stability.
Learning Rate Strategy
Start with a higher learning rate than you might use for plain networks—ResNets' improved gradient flow allows this. A common approach:
- Initial LR: 0.1 for batch size 256
- Warmup: Gradually increase LR over the first few epochs (prevents early instability)
- Step decay: Reduce by 10× at 30%, 60%, and 90% of total epochs
- Or use cosine annealing for smoother decay
Training Strategies
- Longer training: ResNets benefit from extended training (180-200 epochs on ImageNet)
- Weight decay: Use moderate regularization (1e-4) to prevent overfitting
- Data augmentation: Standard crops, flips, and color jittering work well
- Batch size: Larger batches (256+) are common; scale LR proportionally
Think of it like breaking in a new car—you need a careful warm-up period, then you can push harder, but consistent maintenance (regularization) keeps it running smoothly.
Key Takeaway: ResNets train best with He initialization, learning rate warmup followed by aggressive schedules, and patient training with strong regularization—their skip connections enable these aggressive strategies that would destabilize plain networks.