Course contentsShow
← Machine Learning and Deep Learning
Lesson 1375 of 3,53831. Multimodal ModelsFree lesson

Early Vision-Language Models: Visual Question Answering

How early VQA models combined CNNs and RNNs to answer questions about images before unified pretraining.

Early Vision-Language Models: Visual Question Answering

What You'll Learn

You'll learn how the first generation of VQA models bridged vision and language by combining CNNs and RNNs before the era of unified pretraining.

The Core Problem

Before models like CLIP and unified transformers, researchers faced a fundamental challenge: how do you make a machine understand both an image and a natural language question, then produce a coherent answer? This required bridging two very different modalities.

The Early Architecture Pattern

Early Visual Question Answering (VQA) models followed a straightforward "two-tower" approach:

Vision Processing

A CNN (typically ResNet or VGG) processed the input image to extract visual features. Think of this as the model's "eyes" — it creates a rich numerical representation of what's in the picture.

Language Processing

An RNN or LSTM encoded the question text into a semantic representation. This is the model's "language understanding" component, capturing what's being asked.

Fusion and Prediction

The visual and text embeddings were then combined (often through simple concatenation or element-wise operations) and fed into a classifier that predicted the answer from a fixed vocabulary.

This pipeline was like having two separate experts — one who sees but doesn't read, and one who reads but doesn't see — whose insights you merge to answer questions.

The Pre-Pretraining Era

Unlike modern models, these components were typically trained separately or from scratch on VQA datasets rather than being pretrained together on massive vision-language data. This limited their ability to learn rich cross-modal alignments.

Key Takeaway: Early VQA models pioneered multimodal understanding by combining independently-trained CNNs and RNNs, establishing the foundational pattern of extract-encode-fuse that modern unified architectures have since improved upon.