Course contentsShow
← Machine Learning and Deep Learning
Lesson 2565 of 3,53855. Self-Supervised LearningFree lesson

Barlow Twins: Redundancy Reduction

Explore Barlow Twins' approach of maximizing cross-correlation between embeddings to learn invariant representations.

Barlow Twins: Redundancy Reduction

You'll learn how Barlow Twins avoids representation collapse by maximizing cross-correlation between embeddings while minimizing redundancy across dimensions.

The Core Problem

Previous self-supervised methods like SimCLR required negative pairs or stop-gradient tricks to prevent collapse. Barlow Twins takes a fundamentally different approach: it directly tackles redundancy in learned representations.

How Barlow Twins Works

Imagine you're teaching two students (twin networks) to describe the same image using different augmented views. Barlow Twins wants their descriptions to be:

  1. Highly correlated for the same dimension (e.g., both students agree on "blueness")
  2. Uncorrelated across different dimensions (e.g., "blueness" shouldn't predict "roundness")

The method computes a cross-correlation matrix between embeddings from two augmented views of the same batch. Ideally, this matrix should look like an identity matrix:

  • Diagonal elements = 1 (perfect agreement on each feature)
  • Off-diagonal elements = 0 (features are independent)

The Loss Function

Barlow Twins' loss has two components:

  • Invariance term: Pushes diagonal elements toward 1 (embeddings agree across views)
  • Redundancy reduction term: Pushes off-diagonal elements toward 0 (dimensions are decorrelated)

This elegant approach naturally prevents collapse: if all dimensions output the same value, off-diagonal correlations would be high, incurring a large penalty.

Why It Matters

Unlike contrastive methods, Barlow Twins doesn't need negative pairs or asymmetric architectures. It learns rich, diverse representations by explicitly enforcing that different feature dimensions capture different aspects of the data.

Key Takeaway: Barlow Twins prevents collapse through redundancy reduction—making embedding dimensions agree across views (invariance) while being decorrelated from each other (diversity).