Course contentsShow
← Machine Learning and Deep Learning
Lesson 1410 of 3,53831. Multimodal ModelsFree lesson

VQA Model Architectures

Learn typical VQA architectures that fuse visual features from CNNs with question embeddings, followed by classification or answer generation heads.

VQA Model Architectures

You'll learn: How Visual Question Answering models combine image understanding and language processing into a unified architecture.

Core Idea

Visual Question Answering (VQA) requires a model to see an image, understand a natural language question about it, and produce an answer. The architecture must fuse information from two completely different modalities: vision and language.

Think of it like a student taking an open-book exam: they need to look at a diagram (visual features) and read the question (text features), then combine both to give the right answer.

Typical VQA Architecture Pipeline

1. Visual Feature Extraction

A CNN or Vision Transformer processes the input image, producing rich visual representations. These might be region features (specific objects/areas) or global image embeddings.

2. Question Encoding

The question text goes through an embedding layer and encoder (like LSTM, GRU, or Transformer), creating a question representation that captures semantic meaning.

3. Multimodal Fusion

This is where the magic happens. The visual and text features are combined through:

  • Concatenation followed by fully connected layers
  • Element-wise multiplication or addition
  • Cross-modal attention where the question "attends" to relevant image regions (similar to attention in captioning)

4. Answer Prediction Head

The fused representation feeds into a classifier (for multiple-choice answers) or a generation module (for open-ended answers). Most VQA datasets treat it as classification over a fixed vocabulary of common answers.

Key Takeaway: VQA architectures fuse CNN-extracted visual features with text embeddings through attention or concatenation, then use a classification head to select the most likely answer from a predefined vocabulary.