Course contentsShow
← Machine Learning and Deep Learning
Lesson 2551 of 3,53855. Self-Supervised LearningFree lesson

Projection Head Design and Representation Quality

Understand why adding an MLP projection head before contrastive loss improves learned representations.

Projection Head Design and Representation Quality

What You'll Learn

You'll understand why adding a simple MLP projection head before the contrastive loss dramatically improves the quality of learned representations.

The Puzzle of Projection Heads

In SimCLR (and most contrastive learning frameworks), the encoder produces representations h, but we don't apply the contrastive loss directly to h. Instead, we add a small MLP—called a projection head—that transforms h into z, and compute the InfoNCE loss on z.

Here's the counterintuitive part: we throw away the projection head after training and use only the encoder's h representations for downstream tasks. Why does this detour improve h?

Why It Works

The projection head creates a buffer zone that protects the encoder's representations. Here's the insight:

The contrastive loss is destructive by design. It forces representations to collapse similar views together and push different examples apart. This aggressive optimization can discard information that's useful for downstream tasks but not immediately necessary for distinguishing augmented pairs.

By applying the loss to z instead of h, the projection head absorbs this information loss. The encoder's h can retain richer, more general features because the projection head does the "dirty work" of satisfying the contrastive objective.

Analogy: Think of h as your long-term memory and z as scratch paper. The projection head lets you work out the contrastive problem on scratch paper without permanently erasing valuable long-term knowledge.

Design Patterns

SimCLR uses a 2-layer MLP with ReLU activation. Empirically, this simple design works remarkably well—deeper or more complex heads don't help much, and removing the head entirely hurts performance significantly.

Key Takeaway: A non-linear projection head between the encoder and contrastive loss improves representation quality by decoupling the encoder's features from the specific demands of the contrastive objective, preserving information useful for diverse downstream tasks.