Course contentsShow
← Machine Learning and Deep Learning
Lesson 2540 of 3,53855. Self-Supervised LearningFree lesson

The Importance of Large Batch Sizes

Why contrastive learning benefits from large batches that provide more negative samples per positive pair.

The Importance of Large Batch Sizes

What you'll learn: Why contrastive learning performs better when you use large batch sizes during training.

The Problem with Small Batches

In contrastive learning, your model learns by pulling positive pairs (augmented versions of the same image) together while pushing negative pairs (different images) apart. The InfoNCE loss we learned earlier requires comparing your positive pair against many negative samples to work effectively.

When you use a small batch size—say 32 images—you only get 31 negative samples for each positive pair (all the other images in the batch). This gives your model a limited view of what "different" looks like.

Why Bigger is Better

Think of it like learning to recognize faces: if you only compare someone to 5 other people, you might think their round glasses are unique. But compare them to 1,000 people, and you'll learn that glasses are common—you need subtler features to distinguish them.

Large batch sizes (512, 1024, or even 8192 in frameworks like SimCLR) provide more negative samples per positive pair. This forces your model to:

  • Learn more discriminative representations
  • Better separate similar-but-different examples
  • Avoid trivial solutions (like focusing on color alone)

The Mathematical View

Remember the InfoNCE loss denominator sums over all negatives. With more negatives, the gradient signal becomes richer and more stable, leading to representations that generalize better.

Key Takeaway: Contrastive learning thrives on large batch sizes because more negative samples create a harder, more informative learning task that produces superior feature representations—though this requires significant GPU memory.