Course contentsShow
← Machine Learning and Deep Learning
Lesson 2555 of 3,53855. Self-Supervised LearningFree lesson

Momentum Update Strategy

Learn how the momentum encoder slowly tracks the online encoder to maintain representation consistency.

Momentum Update Strategy

What you'll learn: How the momentum encoder in self-supervised learning smoothly tracks the online encoder to maintain stable, consistent representations over time.

The Core Challenge

In MoCo (Momentum Contrast), we maintain two encoders: an online encoder (updated via backpropagation) and a momentum encoder (used to encode negatives in the queue). If we updated both identically, the momentum encoder would change too rapidly, causing the representations in our queue to become inconsistent and degrading contrastive learning quality.

How Momentum Update Works

Instead of copying weights directly, the momentum encoder slowly tracks the online encoder using an exponential moving average:

θ_momentum ← m · θ_momentum + (1 - m) · θ_online

Where:

  • θ_momentum = momentum encoder parameters
  • θ_online = online encoder parameters
  • m = momentum coefficient (typically 0.999)

The Slow-Follower Analogy

Think of two hikers: the online encoder is exploring quickly, trying different paths (learning from gradients). The momentum encoder is a careful follower who averages the leader's positions over time, smoothing out erratic movements and maintaining a stable trajectory.

With m = 0.999, the momentum encoder only updates by 0.1% toward the current online encoder each step. This creates:

  • Stability: Queue representations remain relatively consistent
  • Smooth evolution: The encoder evolves gradually, not abruptly
  • Better representations: Avoids the collapse issues from rapidly changing encoders

Why This Matters

Without momentum updates, identical negative samples in the queue would have wildly different representations depending on when they were encoded, breaking the contrastive learning objective. The slow tracking ensures representation consistency across the queue.

Key Takeaway: The momentum update strategy uses exponential moving averages to make the momentum encoder slowly follow the online encoder, ensuring stable and consistent representations for negative samples in the contrastive learning queue.