Type I and Type II errors are the statistician’s names for the two ways a yes-or-no decision goes wrong. The terms come from hypothesis testing, where you start from a null hypothesis, the default claim that nothing interesting is happening, and ask whether the data give enough reason to reject it.
- A Type I error rejects the null when it is actually true: you declare an effect that isn’t there. In classification terms, a false positive.
- A Type II error fails to reject the null when it is actually false: you miss an effect that is there. A false negative.
The two knobs
The risk of a Type I error, written α, is chosen in advance as the significance level of the test; 5% is a common choice. The risk of a Type II error, written β, is not chosen directly. As the NIST handbook puts it, it depends on the size of the real discrepancy: large effects are easy to detect, small ones are easy to miss. It also rises as α is made stricter, so the two risks pull against each other. Power, 1 − β, is the chance of catching an effect that really exists, and it rises with sample size. In an A/B test, too few users means low power: a real improvement can come back “not significant”.
In machine learning
A classifier makes the same trade every time you set its decision threshold. Lowering the bar for “positive” catches more real cases, cutting Type II errors and raising recall, at the price of more false alarms, which lowers precision. A confusion matrix counts both kinds of mistake side by side.
Which error is worse depends on the stakes. Missing a tumour usually costs more than a second look at a healthy scan, while a spam filter that bins a real invoice may be worse than one that lets some spam through. That judgment belongs to the people deploying the model, not to the metric.