VLM Architectures: CLIP, BLIP, and Flamingo
You'll learn: How vision-language models combine vision encoders with language models through three foundational architectural patterns.
Core Architectural Patterns
Vision-language models must bridge two fundamentally different domains: pixel-level visual data and discrete token-based language. Three landmark architectures pioneered distinct approaches to this challenge.
CLIP: Contrastive Dual-Encoder
CLIP (Contrastive Language-Image Pre-training) uses separate encoders for vision and text that project into a shared embedding space. Think of it like two translators converting different languages into the same universal language. The model learns by contrasting matching image-text pairs against mismatched ones during training, creating aligned representations without explicit supervision.
Use case: Zero-shot image classification, semantic search, image retrieval.
BLIP: Encoder-Decoder with Bootstrapping
BLIP (Bootstrapped Language-Image Pre-training) extends the dual-encoder pattern with a text decoder for generation tasks. It uses three objectives: image-text contrastive learning (like CLIP), image-text matching (verifying if pairs belong together), and image-conditioned language modeling (generating captions). The bootstrapping technique filters noisy web data by generating synthetic captions and filtering them.
Use case: Image captioning, visual question answering, image-text retrieval.
Flamingo: Few-Shot Perceiver Architecture
Flamingo introduces cross-attention layers that inject visual information directly into a frozen pre-trained language model. Images are encoded and passed to the language model through gated cross-attention mechanisms, allowing the model to "perceive" images at relevant points in text generation. This architecture enables few-shot learning—understanding new visual concepts from just a few examples.
Use case: Multi-image reasoning, conversational AI with visual context, few-shot visual tasks.
Common Thread
All three architectures share the fundamental pattern: encode images → align with language → enable downstream tasks. They differ in how tightly vision and language are coupled and what tasks they prioritize.
Key Takeaway: CLIP aligns embeddings contrastively, BLIP adds generation capabilities with bootstrapped data, and Flamingo interleaves vision into language models for few-shot reasoning—each architecture optimizes for different multimodal interaction patterns.