Course contentsShow
← Machine Learning and Deep Learning
Lesson 908 of 3,53821. CNN ArchitecturesFree lesson

Identity vs Projection Shortcuts

When to use identity mappings versus 1x1 convolutions to match dimensions in skip connections.

Identity vs Projection Shortcuts

What you'll learn: When to use identity mappings versus 1×1 convolutions to match dimensions in skip connections.

The Dimension Matching Problem

In residual networks, skip connections add the input directly to the output: output = F(x) + x. But what happens when the input and output have different dimensions? You can't add a 64-channel tensor to a 128-channel tensor—the shapes must match perfectly.

Two Types of Shortcuts

Identity Shortcuts

The simplest solution: when dimensions already match, just pass the input through unchanged. No parameters, no computation—literally just x being added to F(x). This is like a direct bridge that requires no modification.

When to use: The input and output have the same spatial dimensions (height, width) and the same number of channels.

Projection Shortcuts

When dimensions don't match, use a 1×1 convolution (with appropriate stride) to transform the input to match the output shape. This "projects" the input into the new dimensional space.

When to use:

  • When spatial dimensions change (e.g., downsampling with stride 2)
  • When the number of channels increases (e.g., 64 → 128 channels)

Think of it as a custom adapter: you need to reformat the input so it can be added to the transformed output.

Design Trade-off

Identity shortcuts are parameter-free and preserve gradient flow perfectly—ResNet uses them whenever possible. Projection shortcuts add learnable parameters but are necessary when dimensions change. Modern ResNets use projections sparingly, typically only at the beginning of each residual stage where spatial size halves and channels double.

Key Takeaway: Use identity shortcuts when dimensions match (no extra parameters, perfect gradient flow); use projection shortcuts (1×1 convolutions) only when you must transform dimensions to make the addition possible.