Landmark 2 stops to get here · leads to 2

Test Set

Data locked away until the end of a project and used once to estimate how the finished model will perform on new, unseen examples.

Your route here

2 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Train-Test Split ✓ understood

    Dividing a dataset into separate portions for training the model and evaluating its performance on unseen data.

  3. Test Set · you are here ✓ understood

Picture it

  1. 01 Split the data Lock the test set away first
  2. 02 Train on training set Fit the model's parameters
  3. 03 Tune on validation set Pick hyperparameters and checkpoints
  4. 04 Evaluate on test set once An unbiased estimate on unseen data
Notice that the test set is touched only at the very end; using it earlier leaks information and makes the final score optimistic.

A test set is the data you hold back to answer one question at the end: how well will this model do on examples it has never seen? It plays no part in training and no part in choosing the model. Its value comes entirely from that separation, and it’s gone the moment the test set starts influencing decisions.

It’s the last of the three splits made by a train-test split. The training set fits the weights, the validation set picks hyperparameters and checkpoints, and the test set reports the result.

Why it has to stay untouched

The flow in the diagram is really a rule about information. Anything you look at and then act on becomes, in effect, training data. Check test accuracy, tweak the model, check again, and you’re tuning on the test set: the final number turns optimistic, because it rewards choices that happened to suit those particular examples.

Data leakage is the accidental version. Duplicate examples across splits, preprocessing statistics computed on the full dataset, or future information in time-series features all let test data seep into training.

Size and shape

A test set should look like the data the model will face in use. For time-ordered data, that means testing on a later period than you trained on. It also has to be big enough that the score isn’t mostly noise: the number of test examples sets how precise the estimate is. Accuracy of 90% measured on 100 examples carries a standard error of about 3 percentage points; on 10,000 examples it’s about 0.3. When data is scarce, cross-validation can stand in for a fixed validation split, but a final held-out test is still the cleanest estimate.

The catch

Public benchmarks are shared test sets, and the Deep Learning textbook notes that when one is reused to compare methods for years, evaluations on it become optimistic too: the worry is that the whole field slowly overfits it. Recht and colleagues tested that in 2019 by building fresh test sets for CIFAR-10 and ImageNet, closely following the original collection process. Accuracy fell 3–15% on CIFAR-10 and 11–14% on ImageNet. Progress was real, though: accuracy gains on the original test sets translated into larger gains on the new ones, and the authors traced the drop to the new images being slightly harder rather than to test-set reuse. Either way, a test score describes one sample of data, and a small shift in that sample moves the number.

Where it sits

Before this

Train-Test Split
Test Set

Explore nearby

Sources

  1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; sections 5.3 (test-set reuse) and 5.4.3 (standard error of the test estimate)
  2. Recht, Roelofs, Schmidt and Shankar, "Do ImageNet Classifiers Generalize to ImageNet?" . 2019