Course contentsShow
← Machine Learning and Deep Learning
Lesson 1398 of 3,53831. Multimodal ModelsFree lesson

Prompt Engineering for CLIP

Template design and ensembling multiple prompts significantly improves zero-shot performance.

Prompt Engineering for CLIP

What you'll learn: How carefully designing text prompts and combining multiple prompt variations dramatically improves CLIP's zero-shot classification accuracy.

The Power of Prompt Design

When you learned about zero-shot classification with CLIP, you probably used simple class names like "cat" or "dog" as text inputs. But CLIP was trained on full sentences and captions from the internet, not isolated words. This creates a distribution mismatch: single words don't match the natural language CLIP saw during training.

Template design solves this by wrapping class names in natural sentences. Instead of just "cat", you use "a photo of a cat" or "a picture of a cat". This small change aligns better with CLIP's training data and consistently improves accuracy.

Ensembling Multiple Prompts

Here's where it gets powerful: different phrasings can capture different aspects of a concept. For example:

  • "a photo of a {class}"
  • "a blurry photo of a {class}"
  • "a painting of a {class}"

By creating multiple prompt templates and averaging their text embeddings, you create a more robust representation of each class. Think of it like asking several experts to describe the same concept—the consensus view is often more reliable than any single perspective.

Research shows that ensembling 7-10 carefully chosen prompts can improve zero-shot accuracy by 3-5 percentage points across diverse datasets. The templates should cover common variations: different media types, viewpoints, and contexts where the class might appear.

Why It Works

CLIP learns associations between images and how people naturally describe them online. Good prompts tap into those learned patterns. Poor prompts (like bare class names) force CLIP to work outside its comfort zone.

Key Takeaway: Template design and prompt ensembling bridge the gap between your classification task and CLIP's training distribution, unlocking significantly better zero-shot performance without any fine-tuning.