CLIP's Image Encoder
What You'll Learn
You'll understand how CLIP transforms images into feature vectors that can be compared with text representations in a shared embedding space.
The Image Processing Component
CLIP's image encoder is the vision half of the contrastive learning system. Its job is to take an input image and convert it into a fixed-dimensional feature vector (embedding) that captures the semantic meaning of that image.
CLIP offers two architectural choices for this encoder:
Vision Transformer (ViT) Backbone
The default and more modern approach uses a Vision Transformer. The image is split into patches, processed through patch embeddings and positional encodings, then fed through transformer encoder layers with multi-head self-attention. The final representation typically comes from the class token (CLS token) output, which aggregates information from all patches.
ResNet Backbone
Alternatively, CLIP can use a modified ResNet (a CNN architecture). The convolutional layers extract hierarchical features, and the final pooled representation serves as the image embedding. This option trades some performance for faster inference.
Why This Matters
Both encoder options produce embeddings in the same dimensional space as the text encoder—typically 512 or 768 dimensions. This shared embedding space is what enables CLIP to compare images and text directly using contrastive loss. The image encoder must learn to capture visual semantics that align with how language describes those visuals.
Think of it like translating images into a universal language that both vision and text can "speak."
Key Takeaway: CLIP's image encoder (Vision Transformer or ResNet) converts images into fixed-dimensional embeddings that live in the same semantic space as text embeddings, enabling direct vision-language comparison.