VICReg: Variance-Invariance-Covariance Regularization
You'll learn: How VICReg prevents representation collapse using three explicit regularization terms that shape embeddings without requiring negative pairs.
The Core Idea
VICReg (Variance-Invariance-Covariance Regularization) avoids collapse in self-supervised learning by explicitly enforcing three properties on learned representations:
- Invariance: Embeddings from different augmented views of the same image should be similar
- Variance: Each dimension of the embedding should maintain high variance across the batch (prevents all embeddings from collapsing to a constant)
- Covariance: Different embedding dimensions should be decorrelated (prevents dimensions from encoding redundant information)
Think of it like organizing a library: you want books about the same topic close together (invariance), you want books spread across shelves not piled in one corner (variance), and you want each shelf to specialize in different subjects rather than duplicating content (covariance).
The Three Loss Terms
VICReg's total loss is a weighted sum:
Invariance term: Standard MSE between embeddings of two augmented views, making representations consistent.
Variance term: Penalizes when the standard deviation of any embedding dimension (computed across the batch) falls below a threshold (typically 1.0), ensuring embeddings don't collapse to a single point.
Covariance term: Penalizes off-diagonal elements of the covariance matrix computed from batch embeddings, encouraging different dimensions to capture independent features.
Unlike contrastive methods (SimCLR, MoCo) that need negative pairs, or methods relying on asymmetry (BYOL, SimSiam), VICReg uses explicit mathematical constraints that directly prevent the trivial solution where all embeddings become identical.
Key Takeaway: VICReg prevents collapse through three explicit regularization terms—invariance keeps augmented views similar, variance spreads embeddings apart, and covariance decorrelates dimensions—offering a mathematically transparent alternative to contrastive or asymmetric methods.