Course contentsShow
← Machine Learning and Deep Learning
Lesson 1392 of 3,53831. Multimodal ModelsFree lesson

CLIP Architecture Overview

Dual encoder design: separate image and text encoders producing aligned embeddings in shared space.

CLIP Architecture Overview

What you'll learn: How CLIP uses two separate encoders to map images and text into a shared embedding space where similar pairs align.

The Dual Encoder Design

CLIP (Contrastive Language–Image Pre-training) solves the vision-language alignment problem through an elegant dual encoder architecture. Unlike previous models that fuse vision and language features through complex cross-attention, CLIP keeps them separate—then brings them together in a shared embedding space.

Think of it like two translators working independently: one converts images into a universal language, another converts text into that same language. Once translated, you can directly compare them because they "speak" the same language.

How It Works

CLIP consists of two independent encoders:

  1. Image Encoder: Processes images (originally a Vision Transformer or ResNet) and outputs a fixed-size embedding vector
  2. Text Encoder: Processes text captions (a Transformer) and outputs a matching-size embedding vector

Both encoders project their outputs into the same dimensional space (e.g., 512 or 768 dimensions). The key innovation: CLIP trains these encoders so that an image embedding sits close to its correct text description's embedding, and far from unrelated descriptions.

This alignment happens through contrastive learning on massive image-text pairs from the internet. The model learns to maximize similarity between matching pairs while minimizing similarity between mismatched ones.

Why This Matters

The dual encoder design enables efficient zero-shot transfer: you can encode images and text independently, then compare them via simple dot products. No complex cross-attention needed at inference time, making CLIP remarkably fast and flexible for tasks like image classification, retrieval, and more.

Key Takeaway: CLIP's dual encoder architecture separately processes images and text through independent encoders, then aligns their outputs in a shared embedding space where semantically similar pairs cluster together.