A validation set is a slice of labeled data kept out of training and used to make decisions about the model: which learning rate, how many layers, how strong the regularization, when to stop. The model never fits its weights to these examples, so its score on them is a fair preview of how it will do on new data, at least until you lean on it too hard.
It’s the middle of the three splits made by a train-test split. The training set fits the parameters. The validation set picks among the models that training produced. The test set is opened once, at the end, to report the result.
Why training data can’t do this job
Some settings, called hyperparameters, can’t be chosen on the training data, because the training data always votes for more capacity. A bigger model or a weaker penalty will fit the training examples better, so choosing on training error would pick the most overfit option every time. Held-out examples break the tie. Goodfellow, Bengio and Courville’s textbook describes the usual recipe: set aside about 20 percent of the training data for validation and train on the other 80.
What the diagram shows
The curves are what you watch during a training run. Training loss falls almost steadily, because the optimizer is minimizing it directly. Validation loss falls with it at first, then flattens and climbs once the model starts fitting noise. The lowest point on the validation curve is the checkpoint worth keeping, and saving it automatically is exactly what early stopping does. The same comparison, repeated across different settings, drives hyperparameter tuning.
When data is scarce
A small validation set gives a noisy score, and a lucky split can crown the wrong model. Cross-validation fixes this by rotating: split the data into k folds, hold each one out in turn, and average the k scores. It costs k training runs, which is why large deep-learning projects usually settle for one fixed validation split.
The catch
Every decision made on the validation set leaks a little information about it into the model. Try enough configurations and the best validation score is partly luck, fitted to those particular examples. The textbook says so directly: because the validation set is used to “train” the hyperparameters, its error underestimates the true generalization error, though usually by less than training error does. That’s why the number you report should come from a test set that took no part in any choice.