SimSiam: Simple Siamese Networks
What you'll learn: How SimSiam achieves self-supervised learning using an elegantly simple stop-gradient operation instead of complex momentum encoders.
The Surprising Simplicity
After seeing BYOL's momentum encoder and MoCo's queue mechanism, you might think self-supervised learning requires sophisticated components to avoid collapse. SimSiam proves otherwise: it uses identical twin networks (Siamese architecture) with just a stop-gradient operation—no momentum encoder, no negative pairs, no large batches required.
How SimSiam Works
SimSiam takes two augmented views of the same image and processes them through:
- An encoder (shared backbone)
- A projection head (shared MLP)
- A prediction head (only on one branch)
The key insight: maximize similarity between the predicted representation of view 1 and the stop-gradient representation of view 2, then vice versa:
loss = -cosine_similarity(predictor(z1), stop_gradient(z2))
-cosine_similarity(predictor(z2), stop_gradient(z1))
The stop_gradient operation prevents gradients from flowing back through that branch during backpropagation.
Why It Doesn't Collapse
The asymmetry created by the predictor head and stop-gradient breaks the trivial solution. Think of it like a student (predictor) trying to match a frozen snapshot of their past self—they must learn meaningful patterns rather than just outputting zeros.
Unlike BYOL, SimSiam uses no momentum—both views pass through the exact same encoder weights. The stop-gradient alone provides enough asymmetry to prevent collapse.
Key Takeaway: SimSiam demonstrates that self-supervised learning can be remarkably simple—just Siamese networks, a predictor head, and a stop-gradient operation are sufficient to learn high-quality representations without momentum encoders or negative pairs.