ViLBERT: Dual-Stream Vision-Language Architecture
What You'll Learn
You'll understand how ViLBERT processes images and text in separate specialized streams that communicate through cross-modal attention layers.
The Dual-Stream Approach
Most early vision-language models tried to mash images and text together into one big encoder. ViLBERT (Vision-and-Language BERT) takes a different approach: it maintains two separate streams—one for vision, one for language—that run in parallel but talk to each other at key points.
Think of it like two experts working on the same project: a visual analyst and a language specialist. Each focuses on their domain, but they regularly share insights through meetings (cross-modal attention layers).
How It Works
The Vision Stream processes image regions (extracted from object detectors) through transformer layers, building rich visual representations.
The Language Stream processes text tokens through transformer layers, just like BERT, understanding linguistic context.
Cross-Modal Attention Layers are inserted at regular intervals. Here's where the magic happens:
- The vision stream can "query" the language stream: "What text relates to this image region?"
- The language stream can "query" the vision stream: "What visual content matches this word?"
This bi-directional exchange allows each modality to be informed by the other without losing its specialized processing capability.
Why Separate Streams?
Images and text have fundamentally different structures. Vision needs spatial reasoning; language needs sequential understanding. By keeping streams separate until cross-modal layers, ViLBERT preserves these domain-specific inductive biases while still learning alignments between modalities.
Key Takeaway: ViLBERT uses dual specialized streams (vision and language) connected by cross-modal attention layers, allowing each modality to process information naturally while learning rich multimodal representations through controlled interaction points.