MoCo: Momentum Contrast Framework
What You'll Learn
You'll learn how MoCo solves contrastive learning's memory problem by using a momentum encoder and queue to maintain consistent, large-scale negative samples.
The Challenge: Memory vs Consistency
In contrastive learning, you want to compare your anchor against many negative examples to learn good representations. SimCLR solves this with huge batches, but that requires massive compute. MoCo takes a different approach: what if we could reuse negative samples from previous batches?
The Momentum Encoder Solution
MoCo maintains two encoders:
- Query encoder: Your standard encoder, updated via backpropagation
- Momentum encoder: A slowly-updated copy that encodes negatives
The momentum encoder updates using an exponential moving average of the query encoder's weights:
θ_momentum = m * θ_momentum + (1 - m) * θ_query
where m (e.g., 0.999) is a large momentum coefficient. This slow update keeps the encoder consistent across many batches, so older negatives remain comparable to newer ones.
The Queue: A Dictionary of Negatives
MoCo maintains a queue (dictionary) of encoded samples from recent batches. When processing a new batch:
- Compute positives with the query encoder
- Compute negatives from the queue (encoded by the momentum encoder)
- Calculate InfoNCE loss
- Enqueue new samples, dequeue oldest ones
This queue can be very large (65,536 samples) without needing enormous batches!
Why It Works
Think of it like a museum curator: the momentum encoder ensures all artifacts (negatives) are catalogued using a consistent system, even as the cataloguing method slowly evolves. The queue lets you maintain a huge collection without storing everything at once.
Key Takeaway: MoCo decouples batch size from the number of negatives by using a momentum-updated encoder and a sample queue, enabling effective contrastive learning without massive computational resources.