Course contentsShow
← Machine Learning and Deep Learning
Lesson 2561 of 3,53855. Self-Supervised LearningFree lesson

BYOL: Bootstrap Your Own Latent

Explore BYOL's architecture with online and target networks that learns without negative pairs using momentum updates.

BYOL: Bootstrap Your Own Latent

You'll learn how BYOL learns powerful representations without negative pairs by using two networks that teach each other through momentum updates.

The Core Innovation

BYOL (Bootstrap Your Own Latent) revolutionizes self-supervised learning by completely eliminating the need for negative pairs. Unlike SimCLR and MoCo that require contrasting positive pairs against many negatives, BYOL learns from positive pairs alone—yet still avoids the collapse problem where all representations become identical.

Architecture: Online and Target Networks

BYOL uses two parallel networks:

  1. Online network: The main learnable network that processes one augmented view and tries to predict the target network's output
  2. Target network: A slowly-updated copy that processes another augmented view and provides stable targets

The online network has three components: an encoder, a projector, and a predictor (the key difference from contrastive methods). The target network has only an encoder and projector.

How Momentum Updates Work

Instead of backpropagating through the target network, BYOL updates it using an exponential moving average (EMA) of the online network's parameters:

θ_target = τ × θ_target + (1 - τ) × θ_online

where τ (typically 0.99) controls how slowly the target network changes. This creates a stable, slowly-evolving teacher for the online network to learn from.

Why It Avoids Collapse

The predictor in the online network (absent in the target) creates an asymmetry that prevents collapse. The online network must predict the target's representations, but not vice versa. Combined with momentum updates, this asymmetry ensures diverse, useful representations emerge without explicit negative sampling.

Key Takeaway: BYOL achieves state-of-the-art self-supervised learning using only positive pairs through an asymmetric architecture with online/target networks and momentum updates, eliminating the need for large batches or negative samples.