Course contentsShow
← Machine Learning and Deep Learning
Lesson 1414 of 3,53831. Multimodal ModelsFree lesson

From VQA to Generative Multimodal Models

See how VQA evolved into generative tasks handled by multimodal LLMs, and learn the trade-offs between discriminative and generative approaches.

From VQA to Generative Multimodal Models

You'll learn how Visual Question Answering evolved into powerful generative multimodal systems and understand the fundamental shift from discriminative to generative approaches.

The Evolution from VQA to Generation

Traditional VQA models (which you've just learned) are discriminative: they select answers from a fixed vocabulary or choose from multiple-choice options. Think of them like a student picking answer "C" on a test. They learned to map vision + question → answer classification.

But what if we want richer, more flexible responses? This is where generative multimodal models enter. Instead of picking from predefined answers, these models generate text token-by-token, just like language models do—but conditioned on both images and text prompts.

The Architectural Shift

Modern generative multimodal models (like Flamingo, BLIP-2, and GPT-4V) unify multiple tasks:

  • Image captioning (which you learned)
  • VQA
  • Visual reasoning
  • Open-ended dialogue about images

They typically combine:

  1. A frozen vision encoder (often CLIP-based)
  2. Cross-modal attention mechanisms
  3. A powerful language model decoder that generates free-form text

The key innovation? Instead of task-specific heads for VQA, captioning, etc., one unified model handles all vision-language tasks through text generation.

Discriminative vs. Generative Trade-offs

Discriminative VQA:

  • ✅ Fast, efficient classification
  • ✅ Easy to evaluate (exact match)
  • ❌ Limited to predefined answers
  • ❌ Can't explain reasoning

Generative Multimodal:

  • ✅ Flexible, open-ended responses
  • ✅ Can explain and reason in detail
  • ✅ Handles novel questions better
  • ❌ Slower generation
  • ❌ Harder to evaluate objectively
  • ❌ May hallucinate details

Key Takeaway: VQA evolved from discriminative answer selection to generative text production, enabling richer, more flexible multimodal interactions at the cost of increased complexity and evaluation difficulty.