Course contentsShow
← Machine Learning and Deep Learning
Lesson 1425 of 3,53831. Multimodal ModelsFree lesson

Referring and Grounding in Multimodal LLMs

Enabling models to point to image regions and ground language in spatial locations.

Referring and Grounding in Multimodal LLMs

What you'll learn: How multimodal LLMs can point to specific regions in images and connect language descriptions to precise spatial locations.

The Core Problem

Imagine asking someone "Where is the red car?" — they'd naturally point at it. Similarly, referring and grounding enables multimodal LLMs to bridge the gap between language and spatial understanding. Referring means the model can talk about specific objects or regions ("the cat on the left"), while grounding means it can locate those objects in the actual image coordinates.

How It Works

Traditional multimodal LLMs (like those you've learned about with vision encoders and projection layers) understand what's in an image, but not precisely where. Grounding adds spatial awareness:

  1. Bounding Box Outputs: The model learns to predict coordinates (x, y, width, height) alongside text tokens
  2. Region Tokens: Special tokens represent spatial locations, linking language to image patches
  3. Segmentation Masks: More precise pixel-level grounding for complex shapes

Think of it like teaching a child not just to recognize "dog" but to point exactly where the dog is when you say "show me the brown dog near the tree."

Training Approach

Models learn this through datasets pairing descriptions with annotated regions (building on visual grounding concepts from earlier VQA and detection work). The architecture extends your familiar encoder-decoder setup but adds specialized output heads for spatial predictions.

Applications

  • "Circle the damaged area in this medical scan"
  • "Highlight all safety violations in this workplace photo"
  • Interactive image editing through natural language

Key Takeaway: Referring and grounding transforms multimodal LLMs from passive image understanders into spatial reasoners that can precisely locate and reference specific visual elements, bridging the language-vision gap at the coordinate level.