Course contentsShow
← Machine Learning and Deep Learning
Lesson 1382 of 3,53831. Multimodal ModelsFree lesson

LXMERT: Three-Stream Architecture for VL Tasks

Understanding LXMERT's object-relationship-question encoder design and cross-modality layers.

LXMERT: Three-Stream Architecture for VL Tasks

What You'll Learn

You'll understand how LXMERT separates vision, language, and cross-modal reasoning into three specialized encoder streams.

The Three-Stream Design Philosophy

While ViLBERT used a dual-stream architecture (one for vision, one for language), LXMERT (Learning Cross-Modality Encoder Representations from Transformers) introduces a three-stream architecture that explicitly models:

  1. Object-Relationship Encoder (vision stream)
  2. Language Encoder (text stream)
  3. Cross-Modality Encoder (fusion stream)

Think of it like a conversation between two specialists and a translator: the vision expert processes image features, the language expert handles text, and the cross-modality translator helps them understand each other.

How the Streams Work Together

Object-Relationship Encoder: Processes detected objects and their spatial relationships using self-attention. It captures visual scene structure independently before cross-modal interaction.

Language Encoder: Applies self-attention over word tokens (like BERT) to build language representations without visual influence initially.

Cross-Modality Encoder: Here's the key innovation—this stream receives outputs from both unimodal encoders and applies:

  • Cross-attention from language to vision: Each word attends to all objects
  • Cross-attention from vision to language: Each object attends to all words
  • Bidirectional information flow creates aligned representations

The three streams process in parallel initially, then the cross-modality layers fuse information. This design allows each modality to develop strong unimodal representations before alignment—crucial for tasks like visual question answering where you need to reason about "what does the text ask?" and "what does the image show?" before combining them.

Key Takeaway: LXMERT's three-stream architecture separates object relationships, language understanding, and cross-modal fusion into specialized encoders, enabling richer vision-language reasoning than earlier dual-stream models.