Early Vision-Language Models: Visual Question Answering
What You'll Learn
You'll learn how the first generation of VQA models bridged vision and language by combining CNNs and RNNs before the era of unified pretraining.
The Core Problem
Before models like CLIP and unified transformers, researchers faced a fundamental challenge: how do you make a machine understand both an image and a natural language question, then produce a coherent answer? This required bridging two very different modalities.
The Early Architecture Pattern
Early Visual Question Answering (VQA) models followed a straightforward "two-tower" approach:
Vision Processing
A CNN (typically ResNet or VGG) processed the input image to extract visual features. Think of this as the model's "eyes" — it creates a rich numerical representation of what's in the picture.
Language Processing
An RNN or LSTM encoded the question text into a semantic representation. This is the model's "language understanding" component, capturing what's being asked.
Fusion and Prediction
The visual and text embeddings were then combined (often through simple concatenation or element-wise operations) and fed into a classifier that predicted the answer from a fixed vocabulary.
This pipeline was like having two separate experts — one who sees but doesn't read, and one who reads but doesn't see — whose insights you merge to answer questions.
The Pre-Pretraining Era
Unlike modern models, these components were typically trained separately or from scratch on VQA datasets rather than being pretrained together on massive vision-language data. This limited their ability to learn rich cross-modal alignments.
Key Takeaway: Early VQA models pioneered multimodal understanding by combining independently-trained CNNs and RNNs, establishing the foundational pattern of extract-encode-fuse that modern unified architectures have since improved upon.