Attention in VQA: Co-Attention and Bilinear Pooling
What you'll learn: How VQA models use co-attention to simultaneously focus on relevant image regions and question words, and how bilinear pooling fuses these features together.
The Challenge
In Visual Question Answering, the model needs to understand which parts of the image relate to which words in the question. A simple approach would process image and text separately, but this misses important connections. If the question asks "What color is the cat's collar?", the model should focus on both "collar" in the text AND the collar region in the image simultaneously.
Co-Attention: Joint Visual-Textual Focus
Co-attention mechanisms attend to image and question together, letting each modality guide the other's attention. Think of it like two spotlights that coordinate: the question guides where to look in the image, while the image helps determine which question words matter most.
Unlike standard attention (which focuses on one modality), co-attention creates a feedback loop:
- Question features help weight which image regions are important
- Image features help weight which question words are relevant
- These attend to each other iteratively or in parallel
This produces question-aware image features and image-aware question features.
Bilinear Pooling: Fusing Multimodal Features
Once you have attended features from both modalities, you need to combine them. Bilinear pooling captures interactions between vision and language features by computing their outer product, creating a rich joint representation.
Instead of simple concatenation or addition, bilinear pooling multiplies every image feature dimension with every text feature dimension, capturing fine-grained correlations. This helps the model learn complex relationships like "red" (text) + "collar region" (image) → relevant answer.
Key Takeaway: Co-attention lets VQA models jointly attend over image regions and question words to find relevant information in both modalities simultaneously, while bilinear pooling fuses these features by capturing their multiplicative interactions for better answer prediction.