Intersection over Union, or IoU, measures how well two shapes line up. In object detection the two shapes are the model’s predicted bounding box and the hand-labelled ground truth box. Take the area where they overlap and divide it by the total area covered by either one. Identical boxes score 1. Boxes that don’t touch score 0.
The same ratio is known in statistics as the Jaccard index. Computer vision uses it because it’s easy to read, and it doesn’t depend on scale: a box that’s 10 pixels off matters a lot for a small object and very little for a large one, and IoU reflects that.
Why the union matters
Dividing by the union, as the diagram shades it, is what stops a detector from cheating. If you only measured how much of the true box the prediction covered, a huge box drawn over the whole image would score perfectly. The union grows with every stray pixel, so both overshooting and undershooting lower the score.
Turning a score into right or wrong
Detection benchmarks use IoU as a pass mark. The PASCAL VOC challenge counted a prediction as correct only when its IoU with a true box exceeded 0.5, a threshold its authors set deliberately low because hand-drawn boxes are imprecise. Anything below becomes a false positive. The COCO benchmark tightened this: its headline average precision is averaged over ten thresholds from 0.50 to 0.95, which rewards detectors that place boxes precisely, not just roughly.
IoU does other jobs too. Non-maximum suppression uses it to spot duplicate boxes around the same object, and semantic segmentation applies the same ratio to pixel masks instead of rectangles, usually reported as mean IoU across classes.
The catch
IoU is flat at zero for every pair of boxes that don’t overlap, whether they’re nearly touching or on opposite sides of the image. That makes it a poor training signal on its own: a model with a wildly wrong box gets no hint which way to move. Generalized IoU, proposed in 2019, fixes this by also penalizing the empty space inside the smallest box that encloses both. The related Dice coefficient weights the overlap more generously and is often used for medical image segmentation.