Projection Heads
What you'll learn: How adding a simple nonlinear projection on top of your encoder dramatically improves contrastive learning quality.
The Core Idea
In contrastive learning (like SimCLR), you might expect to use your encoder's output directly for computing the InfoNCE loss. Surprisingly, research shows that adding a small nonlinear projection head—typically a 2-3 layer MLP—on top of the encoder, using that output for contrastive loss, then discarding the projection head afterward, produces much better final representations.
Why Does This Work?
Think of it like a "sacrificial layer." The projection head learns to emphasize the specific features needed to distinguish positive pairs from negatives during training. This specialization might be too narrow for downstream tasks. By throwing away the projection head and keeping only the encoder, you retain more general, transferable features.
The projection head creates a separate "contrastive space" optimized for the pretext task, while protecting the encoder's representation space from overfitting to augmentation-specific patterns.
The Architecture Pattern
Input Image
↓
Encoder (e.g., ResNet) → representation h
↓
Projection Head (MLP) → projection z
↓
InfoNCE Loss (computed here with z)
[After training: discard projection head, use h for downstream tasks]
A typical projection head: Linear(2048, 2048) → ReLU → Linear(2048, 128), transforming encoder outputs into a lower-dimensional space where contrastive learning happens.
Key Takeaway: Projection heads are nonlinear transformations applied after the encoder in contrastive learning—they improve representation quality during training but are discarded afterward, letting the encoder learn more general features.