Course contentsShow
← Machine Learning and Deep Learning
Lesson 1421 of 3,53831. Multimodal ModelsFree lesson

Flamingo and In-Context Learning with Images

Few-shot vision-language learning where models learn from image-text examples in the prompt.

Flamingo and In-Context Learning with Images

What you'll learn: How models like Flamingo can learn new vision-language tasks from just a few image-text examples shown in the prompt, without updating their weights.

The Core Idea

Remember how GPT models can perform new tasks by just showing them a few examples in the prompt? Flamingo brings this "few-shot learning" capability to the multimodal world, where both images and text serve as examples.

Instead of fine-tuning a model for each new vision-language task, you simply show Flamingo 2-4 examples of what you want (like "here's a photo of a rare bird species, it's called X"), then ask it to do the same for a new image. The model learns the pattern in-context and generalizes immediately.

How It Works

Flamingo interleaves frozen vision encoders with a powerful frozen language model, connecting them through learned gated cross-attention layers. These special attention mechanisms let the language model "peek" at visual features when needed, but can also ignore them when processing pure text.

During training, Flamingo sees sequences mixing images and text—like webpages with photos and captions. This teaches it to condition its language generation on visual context flexibly.

At inference time, you construct a prompt like:

  • Image 1 + "This is a golden retriever"
  • Image 2 + "This is a poodle"
  • Image 3 + "This is a ___"

The model fills in the blank by understanding the pattern from the visual examples.

Why It Matters

This is a breakthrough for vision-language tasks because you don't need task-specific training data or fine-tuning. You can adapt to new visual concepts, rare objects, or specialized domains just by providing a few examples—much closer to how humans learn.

Key Takeaway: Flamingo demonstrates that large multimodal models can learn new vision-language tasks few-shot, using interleaved image-text examples in the prompt without weight updates—bringing GPT-style in-context learning to the visual domain.