Landmark 6 stops to get here · leads to 2

Semantic Segmentation

Labelling every pixel in an image with a class, such as road, car or sky, so the output is a map of what is where rather than a single tag.

Your route here

6 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Supervised Learning ✓ understood

    Learning from examples paired with the correct answer, so a model can predict answers for new inputs it hasn't seen.

  3. Classification ✓ understood

    A supervised learning task where the model assigns each input to one of a fixed set of categories, such as spam or not spam.

  4. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  5. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  6. Computer Vision ✓ understood

    The field of AI that gets computers to extract meaning from images and video: what is in them, where it is, and how it moves.

  7. Semantic Segmentation · you are here ✓ understood

Picture it

Classification

  • One label for the whole image
  • "This image contains a cat"

Semantic segmentation

  • A class label for every pixel
  • All cat pixels share one label
  • Doesn't separate touching objects

Instance segmentation

  • A label per pixel, plus object identity
  • Cat #1 and cat #2 get separate masks
Notice how semantic segmentation moves from one label per image to one per pixel, but still treats all objects of a class as one region.

Semantic segmentation assigns a class to every pixel in an image. Where classification answers “is there a car in this photo?”, segmentation answers “which exact pixels are car, which are road, which are sky?” The output is an image-sized map, each pixel coloured by its class.

That precision is what makes it useful. A self-driving system needs to know where the drivable road ends, not just that a road exists. A medical tool needs the outline of a tumour to measure its size. A mapping pipeline needs to know which satellite pixels are buildings and which are fields.

How it works

A standard image classifier shrinks the picture step by step into a single prediction, throwing away location along the way. Segmentation networks keep it. A key step came in 2014 with fully convolutional networks, which took existing classification CNNs, replaced their final layers with convolutions, and produced a label map the same size as the input, trained end to end, pixels to pixels.

Most designs since follow an encoder-decoder shape. The encoder compresses the image to capture context: what’s in the scene. The decoder expands it back to full resolution to recover detail: exactly where. U-Net, from 2015, adds skip connections that pass fine detail straight from each encoder stage to the matching decoder stage. It was built for microscopy images and designed to train well from very few annotated examples.

How it’s measured

Pixel accuracy is misleading, because big regions like sky or road dominate the count. The usual metric is mean Intersection over Union: for each class, the overlap between predicted and true pixels divided by their union, averaged across classes. Medical work often reports the closely related Dice coefficient.

The catch

The diagram’s middle column shows the limit. Semantic segmentation knows which pixels are “person”, but two people standing shoulder to shoulder merge into one blob. Telling them apart is instance segmentation. Panoptic segmentation, proposed in 2018, combines the two: every pixel gets a class, and countable objects also get an identity.

Labels are the other cost. Every training image needs a hand-drawn mask traced around every object, far slower to produce than a single image label. That’s why segmentation models usually start from a backbone pre-trained on a larger data set.

Where it sits

Semantic Segmentation

Explore nearby

In the research

All papers →

A paper that builds on Semantic Segmentation .

Sources

  1. Long, Shelhamer and Darrell, "Fully Convolutional Networks for Semantic Segmentation" . 2014; CVPR 2015
  2. Ronneberger, Fischer and Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation" . 2015
  3. Kirillov et al., "Panoptic Segmentation" . 2018; CVPR 2019