Course contentsShow
← Machine Learning and Deep Learning
Lesson 1377 of 3,53831. Multimodal ModelsFree lesson

Vision-Language Pretraining Objectives Overview

Survey of pretraining tasks: masked language modeling, image-text matching, and masked region modeling.

Vision-Language Pretraining Objectives Overview

What you'll learn: The main training tasks used to teach multimodal models to connect visual and textual information during pretraining.

What Are Pretraining Objectives?

Just as BERT learns language patterns through masked word prediction, vision-language models need specialized tasks to learn the connection between images and text. These pretraining objectives teach the model to align visual and textual representations before fine-tuning on downstream tasks.

Think of it like learning a new language alongside photographs: you might practice matching captions to images, filling in missing words while looking at pictures, or identifying which parts of an image correspond to certain words.

Three Core Pretraining Tasks

1. Masked Language Modeling (MLM)

The model sees an image and a caption with some words masked out (e.g., "A [MASK] is jumping over a fence"). Using both the visible words and the image content, it must predict the masked words. This teaches the model to use visual context to understand language.

2. Image-Text Matching (ITM)

Given an image-caption pair, the model predicts whether they actually match or if the caption describes a different image. This is typically a binary classification task that forces the model to learn deep cross-modal alignment—understanding what makes text and images semantically consistent.

3. Masked Region Modeling (MRM)

Similar to MLM but for images: certain visual regions (patches or object features) are masked, and the model must predict properties of those regions using surrounding visual context and the paired text. This teaches visual reasoning grounded in language.

These objectives work together synergistically, creating models that truly understand both modalities and their relationships.

Key Takeaway: Vision-language pretraining uses three complementary objectives—masked language modeling, image-text matching, and masked region modeling—to teach models bidirectional understanding between visual and textual information.