Course contentsShow
← Machine Learning and Deep Learning
Lesson 2564 of 3,53855. Self-Supervised LearningFree lesson

Stop-Gradient and Its Role in Preventing Collapse

Understand how stop-gradient operations create asymmetry and enable learning without contrastive loss or momentum.

Stop-Gradient and Its Role in Preventing Collapse

What You'll Learn

You'll understand how a simple stop-gradient operation creates asymmetry between networks, preventing representation collapse without needing contrastive pairs or momentum encoders.

The Core Idea

In methods like SimSiam and BYOL, two networks process augmented versions of the same image. Naively, they could "collapse" — both outputting constant vectors regardless of input, making loss trivially zero but learning nothing useful.

Stop-gradient prevents this by breaking symmetry: one network's parameters receive gradients during backpropagation, while the other's output is treated as a fixed target (no gradients flow backward through it).

Think of it like a teacher-student relationship where the student learns from the teacher, but the teacher doesn't learn from the student during that particular interaction. This asymmetry forces the networks to maintain informative representations.

How It Works

In SimSiam, you have:

  • An online network (learnable predictor + encoder)
  • A target network (same encoder, different augmentation path)

The loss compares the online network's prediction to the target network's output, but gradients only update the online network. The stop-gradient operation (x.detach() in PyTorch) blocks backpropagation through the target branch.

This asymmetry prevents both networks from agreeing on trivial constant outputs because only one side adapts to minimize the loss. The target network's representations evolve indirectly through shared encoder weights, creating a moving target that encourages meaningful learning.

Why It Matters

Without stop-gradient, both sides would symmetrically minimize distance by collapsing to identical constant vectors. With it, you get stable self-supervised learning without negative pairs, large batches, or momentum encoders — just simple asymmetric gradient flow.

Key Takeaway: Stop-gradient creates asymmetry by blocking gradients through one network branch, preventing collapse and enabling self-supervised learning through simple Siamese architectures without contrastive losses.