BYOL: Bootstrap Your Own Latent
You'll learn how BYOL learns powerful representations without negative pairs by using two networks that teach each other through momentum updates.
The Core Innovation
BYOL (Bootstrap Your Own Latent) revolutionizes self-supervised learning by completely eliminating the need for negative pairs. Unlike SimCLR and MoCo that require contrasting positive pairs against many negatives, BYOL learns from positive pairs alone—yet still avoids the collapse problem where all representations become identical.
Architecture: Online and Target Networks
BYOL uses two parallel networks:
- Online network: The main learnable network that processes one augmented view and tries to predict the target network's output
- Target network: A slowly-updated copy that processes another augmented view and provides stable targets
The online network has three components: an encoder, a projector, and a predictor (the key difference from contrastive methods). The target network has only an encoder and projector.
How Momentum Updates Work
Instead of backpropagating through the target network, BYOL updates it using an exponential moving average (EMA) of the online network's parameters:
θ_target = τ × θ_target + (1 - τ) × θ_online
where τ (typically 0.99) controls how slowly the target network changes. This creates a stable, slowly-evolving teacher for the online network to learn from.
Why It Avoids Collapse
The predictor in the online network (absent in the target) creates an asymmetry that prevents collapse. The online network must predict the target's representations, but not vice versa. Combined with momentum updates, this asymmetry ensures diverse, useful representations emerge without explicit negative sampling.
Key Takeaway: BYOL achieves state-of-the-art self-supervised learning using only positive pairs through an asymmetric architecture with online/target networks and momentum updates, eliminating the need for large batches or negative samples.