Semantic segmentation assigns a class to every pixel in an image. Where classification answers “is there a car in this photo?”, segmentation answers “which exact pixels are car, which are road, which are sky?” The output is an image-sized map, each pixel coloured by its class.
That precision is what makes it useful. A self-driving system needs to know where the drivable road ends, not just that a road exists. A medical tool needs the outline of a tumour to measure its size. A mapping pipeline needs to know which satellite pixels are buildings and which are fields.
How it works
A standard image classifier shrinks the picture step by step into a single prediction, throwing away location along the way. Segmentation networks keep it. A key step came in 2014 with fully convolutional networks, which took existing classification CNNs, replaced their final layers with convolutions, and produced a label map the same size as the input, trained end to end, pixels to pixels.
Most designs since follow an encoder-decoder shape. The encoder compresses the image to capture context: what’s in the scene. The decoder expands it back to full resolution to recover detail: exactly where. U-Net, from 2015, adds skip connections that pass fine detail straight from each encoder stage to the matching decoder stage. It was built for microscopy images and designed to train well from very few annotated examples.
How it’s measured
Pixel accuracy is misleading, because big regions like sky or road dominate the count. The usual metric is mean Intersection over Union: for each class, the overlap between predicted and true pixels divided by their union, averaged across classes. Medical work often reports the closely related Dice coefficient.
The catch
The diagram’s middle column shows the limit. Semantic segmentation knows which pixels are “person”, but two people standing shoulder to shoulder merge into one blob. Telling them apart is instance segmentation. Panoptic segmentation, proposed in 2018, combines the two: every pixel gets a class, and countable objects also get an identity.
Labels are the other cost. Every training image needs a hand-drawn mask traced around every object, far slower to produce than a single image label. That’s why segmentation models usually start from a backbone pre-trained on a larger data set.