Course contentsShow
← Machine Learning and Deep Learning
Lesson 2553 of 3,53855. Self-Supervised LearningFree lesson

MoCo: Momentum Contrast Framework

Discover how MoCo uses a momentum encoder and queue to maintain consistent representations across updates.

MoCo: Momentum Contrast Framework

What You'll Learn

You'll learn how MoCo solves contrastive learning's memory problem by using a momentum encoder and queue to maintain consistent, large-scale negative samples.

The Challenge: Memory vs Consistency

In contrastive learning, you want to compare your anchor against many negative examples to learn good representations. SimCLR solves this with huge batches, but that requires massive compute. MoCo takes a different approach: what if we could reuse negative samples from previous batches?

The Momentum Encoder Solution

MoCo maintains two encoders:

  1. Query encoder: Your standard encoder, updated via backpropagation
  2. Momentum encoder: A slowly-updated copy that encodes negatives

The momentum encoder updates using an exponential moving average of the query encoder's weights:

θ_momentum = m * θ_momentum + (1 - m) * θ_query

where m (e.g., 0.999) is a large momentum coefficient. This slow update keeps the encoder consistent across many batches, so older negatives remain comparable to newer ones.

The Queue: A Dictionary of Negatives

MoCo maintains a queue (dictionary) of encoded samples from recent batches. When processing a new batch:

  • Compute positives with the query encoder
  • Compute negatives from the queue (encoded by the momentum encoder)
  • Calculate InfoNCE loss
  • Enqueue new samples, dequeue oldest ones

This queue can be very large (65,536 samples) without needing enormous batches!

Why It Works

Think of it like a museum curator: the momentum encoder ensures all artifacts (negatives) are catalogued using a consistent system, even as the cataloguing method slowly evolves. The queue lets you maintain a huge collection without storing everything at once.

Key Takeaway: MoCo decouples batch size from the number of negatives by using a momentum-updated encoder and a sample queue, enabling effective contrastive learning without massive computational resources.