Image Captioning Task Definition and Evaluation
What you'll learn: You'll understand what image captioning is, how it differs from simpler vision tasks, and how to measure caption quality using standard metrics.
What Is Image Captioning?
Image captioning is the task of generating natural language descriptions of visual content. Unlike classification (which outputs a single label like "dog") or object detection (which outputs bounding boxes), image captioning produces complete sentences like "A golden retriever playing fetch in a park."
Think of it as the inverse of image generation from text: instead of creating an image from a description, you're creating a description from an image.
How It Differs from Classification
Classification assigns discrete labels to images. Captioning generates free-form text that describes relationships, actions, attributes, and context. It's a multimodal generation task that requires understanding visual content and translating it into coherent language.
Standard Evaluation Metrics
Since captions are generated text, we need metrics that compare generated sentences to human-written reference captions:
BLEU (Bilingual Evaluation Understudy)
Originally from machine translation, BLEU measures n-gram overlap between generated and reference captions. Higher scores mean more word sequences match.
METEOR
Improves on BLEU by considering synonyms, stemming, and word order. More forgiving of paraphrasing.
CIDEr (Consensus-based Image Description Evaluation)
Designed specifically for captioning. Weighs n-grams by their informativeness across the dataset, prioritizing descriptive words over common ones.
SPICE (Semantic Propositional Image Caption Evaluation)
Evaluates semantic content by comparing scene graphs (objects, attributes, relationships) rather than exact word matches. Correlates well with human judgment.
Key Takeaway: Image captioning generates natural language descriptions of images, requiring multimodal understanding. Metrics like BLEU, METEOR, CIDEr, and SPICE evaluate caption quality by comparing generated text to human references, each emphasizing different aspects of language similarity and semantic accuracy.