Course contentsShow
← Machine Learning and Deep Learning
Lesson 1395 of 3,53831. Multimodal ModelsFree lesson

CLIP's Training Objective

Contrastive learning on image-text pairs: maximize cosine similarity for matched pairs, minimize for mismatches.

CLIP's Training Objective

What you'll learn: How CLIP learns to align images and text by maximizing similarity for matching pairs while pushing mismatched pairs apart.

The Core Idea

CLIP's training objective uses contrastive learning to teach the model which images and texts belong together. Imagine you have a batch of N image-text pairs (like a photo of a dog with the caption "a golden retriever"). CLIP creates an N×N grid of all possible combinations and learns to:

  1. Maximize cosine similarity for the N correct diagonal pairs (real matches)
  2. Minimize cosine similarity for the N²-N incorrect off-diagonal pairs (mismatches)

This is done simultaneously in both directions: image→text and text→image.

How It Works

Given a batch, CLIP's image encoder produces image embeddings and the text encoder produces text embeddings. These embeddings are normalized and compared using cosine similarity (measuring the angle between vectors, not their magnitude).

The model learns a symmetric loss: for each image, the correct text should have the highest similarity score among all N texts in the batch. And for each text, the correct image should rank highest among all N images.

Think of it like a matching game: CLIP is shown many cards face-up and must learn to pair them correctly, getting rewarded when matched pairs are "close" in embedding space and penalized when mismatched pairs are too similar.

This contrastive approach is incredibly data-efficient because each batch of N pairs generates N² training signals (all possible comparisons).

Key Takeaway: CLIP learns vision-language alignment through contrastive learning—pulling matched image-text embeddings together while pushing mismatched pairs apart using cosine similarity as the metric.