From VQA to Generative Multimodal Models
You'll learn how Visual Question Answering evolved into powerful generative multimodal systems and understand the fundamental shift from discriminative to generative approaches.
The Evolution from VQA to Generation
Traditional VQA models (which you've just learned) are discriminative: they select answers from a fixed vocabulary or choose from multiple-choice options. Think of them like a student picking answer "C" on a test. They learned to map vision + question → answer classification.
But what if we want richer, more flexible responses? This is where generative multimodal models enter. Instead of picking from predefined answers, these models generate text token-by-token, just like language models do—but conditioned on both images and text prompts.
The Architectural Shift
Modern generative multimodal models (like Flamingo, BLIP-2, and GPT-4V) unify multiple tasks:
- Image captioning (which you learned)
- VQA
- Visual reasoning
- Open-ended dialogue about images
They typically combine:
- A frozen vision encoder (often CLIP-based)
- Cross-modal attention mechanisms
- A powerful language model decoder that generates free-form text
The key innovation? Instead of task-specific heads for VQA, captioning, etc., one unified model handles all vision-language tasks through text generation.
Discriminative vs. Generative Trade-offs
Discriminative VQA:
- ✅ Fast, efficient classification
- ✅ Easy to evaluate (exact match)
- ❌ Limited to predefined answers
- ❌ Can't explain reasoning
Generative Multimodal:
- ✅ Flexible, open-ended responses
- ✅ Can explain and reason in detail
- ✅ Handles novel questions better
- ❌ Slower generation
- ❌ Harder to evaluate objectively
- ❌ May hallucinate details
Key Takeaway: VQA evolved from discriminative answer selection to generative text production, enabling richer, more flexible multimodal interactions at the cost of increased complexity and evaluation difficulty.