Course contentsShow
← AI Engineering
Lesson 1756 of 1,88642. Multimodal SystemsFree lesson

CLIP and Contrastive Learning

How CLIP learns joint image-text embeddings through contrastive learning and enables zero-shot classification and retrieval.

CLIP and Contrastive Learning

What you'll learn: How CLIP learns to match images and text in a shared semantic space through contrastive learning, enabling zero-shot classification without task-specific training.

The Core Idea

CLIP (Contrastive Language-Image Pre-training) learns to understand images and text simultaneously by training on millions of image-caption pairs from the internet. Instead of learning to predict labels, it learns which images and text descriptions belong together.

Think of it like a matching game: CLIP sees 100 images and 100 captions. For each image, it learns to pick the correct caption from all 100 options, and vice versa. This creates a shared embedding space where semantically similar images and text end up close together.

How Contrastive Learning Works

CLIP uses two encoders: one for images (typically a Vision Transformer or ResNet) and one for text (a Transformer). During training:

  1. Encode: Each image and caption becomes a vector embedding
  2. Compare: CLIP computes similarity scores between all image-text pairs in the batch
  3. Contrast: It maximizes similarity for correct pairs while minimizing similarity for incorrect pairs
  4. Learn: Gradients push matching pairs together and non-matching pairs apart

Zero-Shot Capabilities

Because CLIP learns general visual-semantic relationships rather than fixed categories, you can classify images into new classes it's never seen. Just provide text prompts like "a photo of a cat" or "a photo of a dog" — CLIP compares the image embedding to each text embedding and picks the closest match.

This same mechanism powers image-text retrieval: search images using natural language, or find descriptions for any image.

Key Takeaway: CLIP uses contrastive learning to align image and text embeddings in a shared space, enabling flexible zero-shot classification and retrieval without retraining for specific tasks.