Momentum Update Strategy
What you'll learn: How the momentum encoder in self-supervised learning smoothly tracks the online encoder to maintain stable, consistent representations over time.
The Core Challenge
In MoCo (Momentum Contrast), we maintain two encoders: an online encoder (updated via backpropagation) and a momentum encoder (used to encode negatives in the queue). If we updated both identically, the momentum encoder would change too rapidly, causing the representations in our queue to become inconsistent and degrading contrastive learning quality.
How Momentum Update Works
Instead of copying weights directly, the momentum encoder slowly tracks the online encoder using an exponential moving average:
θ_momentum ← m · θ_momentum + (1 - m) · θ_online
Where:
θ_momentum= momentum encoder parametersθ_online= online encoder parametersm= momentum coefficient (typically 0.999)
The Slow-Follower Analogy
Think of two hikers: the online encoder is exploring quickly, trying different paths (learning from gradients). The momentum encoder is a careful follower who averages the leader's positions over time, smoothing out erratic movements and maintaining a stable trajectory.
With m = 0.999, the momentum encoder only updates by 0.1% toward the current online encoder each step. This creates:
- Stability: Queue representations remain relatively consistent
- Smooth evolution: The encoder evolves gradually, not abruptly
- Better representations: Avoids the collapse issues from rapidly changing encoders
Why This Matters
Without momentum updates, identical negative samples in the queue would have wildly different representations depending on when they were encoded, breaking the contrastive learning objective. The slow tracking ensures representation consistency across the queue.
Key Takeaway: The momentum update strategy uses exponential moving averages to make the momentum encoder slowly follow the online encoder, ensuring stable and consistent representations for negative samples in the contrastive learning queue.