BYOL Training Dynamics and Predictor Role
You'll learn: How BYOL's predictor network prevents representation collapse and why the asymmetry between online and target networks is essential for stable self-supervised learning.
The Collapse Problem Without a Predictor
In BYOL, two networks process different augmented views of the same image: an online network (updated by gradient descent) and a target network (updated via momentum). Without careful design, both networks could collapse to outputting constant representations—learning nothing useful.
The Predictor's Critical Role
BYOL introduces an asymmetric predictor network that sits on top of the online network only. The online network produces a representation, the predictor transforms it, and this transformed output tries to match the target network's representation (which has no predictor).
Why this prevents collapse: The predictor creates an information bottleneck. If the online network tried to output constant values, the predictor would need to somehow produce diverse outputs matching the target network's varying representations—an impossible task. This forces the online network to learn meaningful, diverse features.
Asymmetry is Key
The predictor exists only in the online branch, never in the target branch. This asymmetry means:
- The target network provides a "moving" goal that evolves smoothly (via momentum updates)
- The online network must continuously adapt through the predictor
- The system cannot settle into trivial constant solutions
Think of it like a student (online + predictor) trying to match a teacher's (target) slowly evolving understanding—the student must genuinely learn rather than just memorize one answer.
Key Takeaway: BYOL's predictor network prevents collapse by creating asymmetry between online and target networks, forcing the online network to learn meaningful representations that can be flexibly transformed to match the evolving target representations.