A hyperparameter is a setting you choose before training starts. It controls how the model learns rather than what it learns. The learning rate, the batch size, the number of epochs, how many layers the network has, and how strong the regularization is are all hyperparameters.
The contrast is with parameters, the weights and biases that training adjusts. A useful picture: the parameters are the dish, the hyperparameters are the oven temperature and cooking time. You set those, wait, and taste the result.
Why they aren’t just learned
Some settings, like the number of layers, aren’t things gradient descent can adjust at all. Others could in principle be learned, but not from the training data. Anything that controls model capacity would simply be pushed to the maximum, because a bigger, less constrained model always fits the training set better. That’s a recipe for overfitting. So hyperparameters are judged on data the model never trained on.
How they get chosen
The standard loop: train with one configuration, score the model on a validation set, try another configuration, keep the best. This search is called hyperparameter tuning, and it comes in a few flavours:
- Manual: start from values that worked in similar published work and adjust by feel.
- Grid search: try every combination of a few candidate values per setting.
- Random search: sample combinations at random. A 2012 study showed this usually beats a grid for the same budget, because typically only a few settings matter much, and random samples explore many more values of each.
- Model-guided search: methods like Bayesian optimization use earlier results to pick promising next trials.
The catch
Every trial is a full or partial training run, so tuning is often the most expensive part of a project. For large models it’s rarely affordable at full size, so teams tune on smaller versions and extrapolate, which doesn’t always carry over.
Tuning also leaks information. Try enough configurations against one validation set and the winner will be partly lucky. That’s why a separate test set is kept untouched until the very end.