Momentum Encoders vs Stop-Gradient
What you'll learn: You'll understand the two dominant approaches for creating asymmetry in non-contrastive self-supervised learning and when to use each.
The Core Problem
Non-contrastive methods like BYOL, SimSiam, and DINO need asymmetry between two network branches to avoid collapse. Without it, both networks would output identical constant vectors. Two main strategies emerged to solve this:
Strategy 1: Momentum Encoders
Momentum encoders (used in BYOL, MoCo, DINO) maintain a slowly-updating target network. The target encoder's weights are an exponential moving average of the online encoder:
θ_target = m × θ_target + (1 - m) × θ_online
where m (momentum coefficient) is typically 0.99 or higher. This creates asymmetry because the target network lags behind, providing stable targets during training.
Pros: Smooth, stable learning; explicit control over update speed
Cons: Extra memory for target network; introduces hyperparameter m
Strategy 2: Stop-Gradient
Stop-gradient (used in SimSiam, Barlow Twins) uses identical architectures but simply blocks gradients from flowing through one branch. The predictor on one side learns to match representations from the other, while stop-gradient prevents trivial solutions.
Pros: Simple, no extra memory, no momentum hyperparameter
Cons: More sensitive to architecture choices (requires predictor); less explicit control
The Trade-off
Think of momentum encoders as a "slow teacher" that guides learning with stability, while stop-gradient is like a "frozen mirror" that forces the predictor to do all the work. Momentum encoders are more robust but computationally heavier; stop-gradient is elegant but requires careful architecture design (especially the predictor).
Key Takeaway: Momentum encoders provide stability through slow-moving targets, while stop-gradient achieves asymmetry through gradient blocking—both prevent collapse but differ in complexity and memory requirements.