Course contentsShow
← Machine Learning and Deep Learning
Lesson 2568 of 3,53855. Self-Supervised LearningFree lesson

Momentum Encoders vs Stop-Gradient

Compare the two main strategies for creating asymmetry in non-contrastive methods and their trade-offs.

Momentum Encoders vs Stop-Gradient

What you'll learn: You'll understand the two dominant approaches for creating asymmetry in non-contrastive self-supervised learning and when to use each.

The Core Problem

Non-contrastive methods like BYOL, SimSiam, and DINO need asymmetry between two network branches to avoid collapse. Without it, both networks would output identical constant vectors. Two main strategies emerged to solve this:

Strategy 1: Momentum Encoders

Momentum encoders (used in BYOL, MoCo, DINO) maintain a slowly-updating target network. The target encoder's weights are an exponential moving average of the online encoder:

θ_target = m × θ_target + (1 - m) × θ_online

where m (momentum coefficient) is typically 0.99 or higher. This creates asymmetry because the target network lags behind, providing stable targets during training.

Pros: Smooth, stable learning; explicit control over update speed
Cons: Extra memory for target network; introduces hyperparameter m

Strategy 2: Stop-Gradient

Stop-gradient (used in SimSiam, Barlow Twins) uses identical architectures but simply blocks gradients from flowing through one branch. The predictor on one side learns to match representations from the other, while stop-gradient prevents trivial solutions.

Pros: Simple, no extra memory, no momentum hyperparameter
Cons: More sensitive to architecture choices (requires predictor); less explicit control

The Trade-off

Think of momentum encoders as a "slow teacher" that guides learning with stability, while stop-gradient is like a "frozen mirror" that forces the predictor to do all the work. Momentum encoders are more robust but computationally heavier; stop-gradient is elegant but requires careful architecture design (especially the predictor).

Key Takeaway: Momentum encoders provide stability through slow-moving targets, while stop-gradient achieves asymmetry through gradient blocking—both prevent collapse but differ in complexity and memory requirements.