VQA Model Architectures
You'll learn: How Visual Question Answering models combine image understanding and language processing into a unified architecture.
Core Idea
Visual Question Answering (VQA) requires a model to see an image, understand a natural language question about it, and produce an answer. The architecture must fuse information from two completely different modalities: vision and language.
Think of it like a student taking an open-book exam: they need to look at a diagram (visual features) and read the question (text features), then combine both to give the right answer.
Typical VQA Architecture Pipeline
1. Visual Feature Extraction
A CNN or Vision Transformer processes the input image, producing rich visual representations. These might be region features (specific objects/areas) or global image embeddings.
2. Question Encoding
The question text goes through an embedding layer and encoder (like LSTM, GRU, or Transformer), creating a question representation that captures semantic meaning.
3. Multimodal Fusion
This is where the magic happens. The visual and text features are combined through:
- Concatenation followed by fully connected layers
- Element-wise multiplication or addition
- Cross-modal attention where the question "attends" to relevant image regions (similar to attention in captioning)
4. Answer Prediction Head
The fused representation feeds into a classifier (for multiple-choice answers) or a generation module (for open-ended answers). Most VQA datasets treat it as classification over a fixed vocabulary of common answers.
Key Takeaway: VQA architectures fuse CNN-extracted visual features with text embeddings through attention or concatenation, then use a classification head to select the most likely answer from a predefined vocabulary.