Segmentation Output Format and Evaluation
What You'll Learn
You'll learn how segmentation models output pixel-level predictions and how to measure their accuracy using standard metrics.
The Segmentation Mask: Pixel-by-Pixel Predictions
Unlike object detection which outputs bounding boxes, semantic segmentation produces a segmentation mask—an image where every single pixel is assigned a class label. Think of it as coloring by numbers: each pixel gets colored according to which category it belongs to (e.g., road, sky, person, car).
The output is typically a 2D array matching your input image dimensions, where each value represents a class index. For an image of size 512×512 with 21 classes, your model outputs a 512×512 array with values from 0 to 20.
Evaluation Metrics
Pixel Accuracy
The simplest metric: what percentage of pixels did you classify correctly?
Pixel Accuracy = Correct Pixels / Total Pixels
While intuitive, this metric has a weakness. If 90% of your image is "background," a lazy model could achieve 90% accuracy by predicting everything as background!
Mean Intersection over Union (mIoU)
The gold standard for segmentation evaluation. For each class:
IoU = Area of Overlap / Area of Union
This measures how well predicted regions overlap with ground truth regions. Calculate IoU for each class, then average them—that's your mean IoU (mIoU).
mIoU naturally handles class imbalance better because rare classes contribute equally to the final score. A model must perform well across all classes, not just the dominant ones.
Key Takeaway: Segmentation outputs are spatial masks where every pixel gets a class label; pixel accuracy measures overall correctness, but mean IoU is the preferred metric because it fairly evaluates performance across all classes, even rare ones.