Auxiliary Classifiers for Training Deep Networks
What you'll learn: How GoogLeNet used auxiliary loss heads partway through the network to help gradients flow better during training of very deep models.
The Vanishing Gradient Problem in Deep Networks
When training very deep networks like GoogLeNet (22 layers), gradients computed at the output must travel backward through many layers to update early weights. As they propagate backward, these gradients can become exponentially smaller—sometimes vanishing almost completely. This makes the early layers learn extremely slowly or not at all.
GoogLeNet's Clever Solution
GoogLeNet introduced auxiliary classifiers: additional small classification heads attached to intermediate layers of the network (not just at the end). These auxiliary heads also make predictions and compute their own losses during training.
Think of it like having multiple teachers grading your work at different stages of a project, not just at the final submission. Each checkpoint gives you feedback sooner, helping you correct mistakes before they compound.
How It Works
During training:
- The main classifier at the network's end computes its loss
- Two auxiliary classifiers attached to intermediate layers (after Inception modules 4a and 4d) also compute their losses
- All three losses are combined (with the auxiliary losses typically weighted at 0.3 each)
- Backpropagation now provides gradient signals from multiple points in the network
This ensures that gradients reaching the early layers are stronger and more informative, helping those layers learn effectively despite the network's depth.
Important: At inference time, the auxiliary classifiers are discarded—they're purely training aids.
Key Takeaway: Auxiliary classifiers combat vanishing gradients by injecting additional gradient signals at intermediate network depths, ensuring early layers receive strong learning signals even in very deep architectures.