Course contentsShow
← Machine Learning and Deep Learning
Lesson 1397 of 3,53831. Multimodal ModelsFree lesson

Zero-Shot Classification with CLIP

Classify images by computing similarity to text prompts like 'a photo of a {class}' without fine-tuning.

Zero-Shot Classification with CLIP

What you'll learn: How CLIP classifies images into arbitrary categories without ever being explicitly trained on those specific classes.

The Core Idea

Zero-shot classification means recognizing objects or scenes the model has never seen labeled examples of during training. CLIP achieves this by matching images to text descriptions rather than predicting fixed class labels.

Instead of training a classifier with categories like "cat," "dog," "car," CLIP learns to measure how well an image matches any text prompt. At inference time, you give it candidate text descriptions like "a photo of a cat" or "a photo of a dog", and CLIP computes similarity scores between the image embedding and each text embedding.

How It Works

  1. Create text prompts for each possible class using templates like "a photo of a {class}", "a picture of a {class}", or domain-specific prompts
  2. Encode the image through CLIP's image encoder to get an image embedding
  3. Encode all text prompts through CLIP's text encoder to get text embeddings
  4. Compute similarity (typically cosine similarity) between the image embedding and each text embedding
  5. Pick the highest score as the predicted class

The Magic

Because CLIP was trained on hundreds of millions of image-text pairs from the internet, it learned visual concepts through natural language. If you want to classify medical images, product photos, or satellite imagery—categories it may not have seen—you just describe them in words. The model transfers its broad vision-language understanding to your specific task.

Analogy: Imagine asking someone who's read millions of encyclopedia entries to match a photo to the best description, even if they've never studied that exact topic formally. They use general knowledge to bridge the gap.

Key Takeaway: Zero-shot classification with CLIP transforms image classification from predicting fixed labels to computing similarity between visual and textual representations, enabling classification on arbitrary categories without fine-tuning.