Course contentsShow
← Machine Learning and Deep Learning
Lesson 2566 of 3,53855. Self-Supervised LearningFree lesson

VICReg: Variance-Invariance-Covariance Regularization

Learn VICReg's explicit regularization terms that prevent collapse through variance, invariance, and covariance objectives.

VICReg: Variance-Invariance-Covariance Regularization

You'll learn: How VICReg prevents representation collapse using three explicit regularization terms that shape embeddings without requiring negative pairs.

The Core Idea

VICReg (Variance-Invariance-Covariance Regularization) avoids collapse in self-supervised learning by explicitly enforcing three properties on learned representations:

  1. Invariance: Embeddings from different augmented views of the same image should be similar
  2. Variance: Each dimension of the embedding should maintain high variance across the batch (prevents all embeddings from collapsing to a constant)
  3. Covariance: Different embedding dimensions should be decorrelated (prevents dimensions from encoding redundant information)

Think of it like organizing a library: you want books about the same topic close together (invariance), you want books spread across shelves not piled in one corner (variance), and you want each shelf to specialize in different subjects rather than duplicating content (covariance).

The Three Loss Terms

VICReg's total loss is a weighted sum:

Invariance term: Standard MSE between embeddings of two augmented views, making representations consistent.

Variance term: Penalizes when the standard deviation of any embedding dimension (computed across the batch) falls below a threshold (typically 1.0), ensuring embeddings don't collapse to a single point.

Covariance term: Penalizes off-diagonal elements of the covariance matrix computed from batch embeddings, encouraging different dimensions to capture independent features.

Unlike contrastive methods (SimCLR, MoCo) that need negative pairs, or methods relying on asymmetry (BYOL, SimSiam), VICReg uses explicit mathematical constraints that directly prevent the trivial solution where all embeddings become identical.

Key Takeaway: VICReg prevents collapse through three explicit regularization terms—invariance keeps augmented views similar, variance spreads embeddings apart, and covariance decorrelates dimensions—offering a mathematically transparent alternative to contrastive or asymmetric methods.