CLIP's Text Encoder
What you'll learn: How CLIP processes text prompts into embeddings that can be matched against images.
Understanding CLIP's Text Side
You've seen how CLIP's image encoder transforms pictures into vectors. Now we need the text side of the equation! CLIP's text encoder is the component that takes natural language descriptions—whether simple labels like "a dog" or complex captions like "a golden retriever playing in the snow"—and converts them into embeddings that live in the same vector space as the image embeddings.
Transformer-Based Architecture
CLIP uses a Transformer-based text encoder similar to the language models you've studied before. Think of it like BERT's encoder, but trained specifically for vision-language alignment rather than pure language understanding.
The process works like this:
- Your text prompt is tokenized into subword units
- Token embeddings are created and combined with positional embeddings
- The sequence flows through multiple Transformer encoder layers with multi-head self-attention
- A special representation (typically from a specific token position) is extracted as the final text embedding
Why Transformers for Text?
Transformers excel at capturing semantic meaning and relationships between words. When you write "a photo of a cat sitting on a laptop," the self-attention mechanism understands that "sitting" relates to both "cat" and "laptop," creating a rich contextual representation that can match against the corresponding visual concept.
The key innovation: both text and image encoders are trained together with contrastive learning, forcing their outputs into the same embedding space where semantically similar concepts cluster together.
Key Takeaway: CLIP's text encoder is a Transformer that converts any text prompt into an embedding vector aligned with visual features, enabling flexible zero-shot image understanding through natural language.