Identity vs Projection Shortcuts
What you'll learn: When to use identity mappings versus 1×1 convolutions to match dimensions in skip connections.
The Dimension Matching Problem
In residual networks, skip connections add the input directly to the output: output = F(x) + x. But what happens when the input and output have different dimensions? You can't add a 64-channel tensor to a 128-channel tensor—the shapes must match perfectly.
Two Types of Shortcuts
Identity Shortcuts
The simplest solution: when dimensions already match, just pass the input through unchanged. No parameters, no computation—literally just x being added to F(x). This is like a direct bridge that requires no modification.
When to use: The input and output have the same spatial dimensions (height, width) and the same number of channels.
Projection Shortcuts
When dimensions don't match, use a 1×1 convolution (with appropriate stride) to transform the input to match the output shape. This "projects" the input into the new dimensional space.
When to use:
- When spatial dimensions change (e.g., downsampling with stride 2)
- When the number of channels increases (e.g., 64 → 128 channels)
Think of it as a custom adapter: you need to reformat the input so it can be added to the transformed output.
Design Trade-off
Identity shortcuts are parameter-free and preserve gradient flow perfectly—ResNet uses them whenever possible. Projection shortcuts add learnable parameters but are necessary when dimensions change. Modern ResNets use projections sparingly, typically only at the beginning of each residual stage where spatial size halves and channels double.
Key Takeaway: Use identity shortcuts when dimensions match (no extra parameters, perfect gradient flow); use projection shortcuts (1×1 convolutions) only when you must transform dimensions to make the addition possible.