Course contentsShow
← Machine Learning and Deep Learning
Lesson 2567 of 3,53855. Self-Supervised LearningFree lesson

DINO: Self-Distillation with No Labels

Understand DINO's self-distillation approach using Vision Transformers with centering and sharpening operations.

DINO: Self-Distillation with No Labels

What You'll Learn

You'll understand how DINO uses self-distillation with Vision Transformers, employing centering and sharpening operations to learn powerful visual representations without any labels.

The Core Idea

DINO (Self-Distillation with no labels) takes the momentum-based self-supervised approach we've seen and applies it specifically to Vision Transformers. Instead of contrastive learning, DINO uses knowledge distillation: a student network learns to match the outputs of a teacher network on different augmented views of the same image.

Here's the twist: both student and teacher have identical architectures, and the teacher is updated as an exponential moving average (EMA) of the student's weights — just like in MoCo and BYOL. The student tries to predict what the teacher would output for differently augmented crops.

Preventing Collapse: Centering and Sharpening

Without negative pairs, how does DINO avoid the collapse problem? It uses two clever operations:

Centering: The teacher's outputs are normalized by subtracting a running average of previous batch centers. Think of it like preventing the model from always predicting the same "safe" answer by shifting the origin dynamically.

Sharpening: A low temperature is applied to the teacher's softmax outputs (like we saw in contrastive learning), making the predictions more confident and peaked. The student, however, uses a higher temperature, creating softer targets.

Together, these operations create an asymmetry between teacher and student that prevents mode collapse while encouraging the student to produce discriminative features.

DINO discovered that Vision Transformers trained this way naturally learn to attend to object boundaries and semantically meaningful regions — even emerging properties like object segmentation — all without labels!

Key Takeaway: DINO achieves self-supervised learning through self-distillation between student and teacher Vision Transformers, using centering to prevent trivial solutions and sharpening to enforce confident teacher predictions.