Regularization is anything you do to a learning algorithm to make it generalize better, even at some cost to how well it fits the training data. The standard deep learning textbook defines it as a change meant to reduce generalization error “but not its training error”. It is the main set of tools against overfitting.
The intuition is a preference for simpler explanations. A flexible model can fit its training examples in many ways, most of which pick up noise. Regularization tilts training toward the smoother, simpler fits that are more likely to hold up on new data.
Three ways to do it
The diagram groups the common techniques by how they work.
Penalize large weights. Add a term to the loss that grows with the size of the weights, scaled by a strength setting, λ. L2 regularization adds the sum of squared weights, which shrinks all weights smoothly toward zero. L1 regularization adds the sum of absolute values, which tends to push many weights to exactly zero, giving a sparse model that effectively selects features. Weight decay shrinks the weights a little at every update. For plain SGD it’s equivalent to L2, but not for adaptive optimizers like Adam, which is why AdamW (2017) applies the decay separately.
Inject noise. Dropout switches off a random subset of units at each training step, so the network can’t rely on any single unit and effectively trains many thinned sub-networks at once. Data augmentation varies the inputs, with flips, crops or colour shifts on images, so the model sees each example in many forms.
Stop early. Early stopping tracks loss on a held-out validation set and keeps the checkpoint where it was lowest, before the model starts memorizing.
Other methods count too, such as label smoothing, and so does simply using more data or a smaller model.
Tuning the strength
Regularization strength is a hyperparameter. Too little and the model overfits; too much and it underfits, unable to fit even the training data. The right value is found by trying several and comparing validation scores, which is the bias–variance trade-off in practice.
The catch
Classical theory says regularization is what stops big models from memorizing, but deep networks complicate that story. A 2016 study found that standard image networks could still fit completely random labels with dropout, weight decay and augmentation switched on, and that the same networks with every regularizer turned off still generalized well on real data. Explicit regularizers help, but much of what makes large networks generalize comes from the data, the architecture and the training process itself.