Course contentsShow
← Machine Learning and Deep Learning
Lesson 2562 of 3,53855. Self-Supervised LearningFree lesson

BYOL Training Dynamics and Predictor Role

Understand how BYOL's predictor network prevents collapse and why asymmetry between networks is crucial.

BYOL Training Dynamics and Predictor Role

You'll learn: How BYOL's predictor network prevents representation collapse and why the asymmetry between online and target networks is essential for stable self-supervised learning.

The Collapse Problem Without a Predictor

In BYOL, two networks process different augmented views of the same image: an online network (updated by gradient descent) and a target network (updated via momentum). Without careful design, both networks could collapse to outputting constant representations—learning nothing useful.

The Predictor's Critical Role

BYOL introduces an asymmetric predictor network that sits on top of the online network only. The online network produces a representation, the predictor transforms it, and this transformed output tries to match the target network's representation (which has no predictor).

Why this prevents collapse: The predictor creates an information bottleneck. If the online network tried to output constant values, the predictor would need to somehow produce diverse outputs matching the target network's varying representations—an impossible task. This forces the online network to learn meaningful, diverse features.

Asymmetry is Key

The predictor exists only in the online branch, never in the target branch. This asymmetry means:

  • The target network provides a "moving" goal that evolves smoothly (via momentum updates)
  • The online network must continuously adapt through the predictor
  • The system cannot settle into trivial constant solutions

Think of it like a student (online + predictor) trying to match a teacher's (target) slowly evolving understanding—the student must genuinely learn rather than just memorize one answer.

Key Takeaway: BYOL's predictor network prevents collapse by creating asymmetry between online and target networks, forcing the online network to learn meaningful representations that can be flexibly transformed to match the evolving target representations.