Course contentsShow
← Machine Learning and Deep Learning
Lesson 2541 of 3,53855. Self-Supervised LearningFree lesson

Momentum Encoders and Memory Banks

Techniques to maintain consistent negative samples across training steps without recomputing all embeddings.

Momentum Encoders and Memory Banks

What you'll learn: How to maintain consistent negative samples across training steps without constantly recomputing embeddings for your entire dataset.

The Problem with Large-Scale Contrastive Learning

In contrastive learning, you need many negative samples to learn good representations. With large batch sizes, you get more negatives per step—but there's a limit to how large batches can grow due to memory constraints. Recomputing embeddings for thousands of negative samples every step is also computationally expensive.

Enter Memory Banks and Momentum Encoders

Memory banks store previously computed embeddings from past batches, letting you access thousands of negatives without recomputing them. Think of it as a "library" of representations you can continuously sample from.

Momentum encoders are a clever solution to keep these stored embeddings consistent. Instead of updating the encoder with standard backpropagation, a momentum encoder updates its weights slowly as a moving average of your main encoder:

θ_momentum = m × θ_momentum + (1 - m) × θ_online

where m (typically 0.999) controls how slowly it changes. This produces stable embeddings for your memory bank—the representations don't shift wildly between steps.

Why This Matters

By combining a momentum encoder with a memory bank, you effectively get access to tens of thousands of consistent negative samples without massive batch sizes or constant recomputation. The momentum encoder ensures embeddings in your bank remain compatible with your current model, while the memory bank provides the scale needed for effective contrastive learning.

Key Takeaway: Momentum encoders update slowly via moving averages to produce stable embeddings, while memory banks store these embeddings across batches, enabling large-scale contrastive learning without prohibitive computational costs.