Landmark 3 stops to get here · leads to 3

Regularization

Any change to training meant to make a model generalize better rather than fit its training data more closely, such as weight penalties, dropout or early stopping.

Your route here

3 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training Data ✓ understood

    The examples a model learns its weights from, kept separate from the validation and test data used to check how well it generalizes.

  3. Overfitting ✓ understood

    When a model fits its training data too closely, noise included, so it scores well on examples it has seen and poorly on new ones.

  4. Regularization · you are here ✓ understood

Picture it

Penalize weights

  • L2 adds λ·Σw² to the loss
  • L1 adds λ·Σ|w|, pushing weights to zero
  • Weight decay shrinks weights each step

Inject noise

  • Dropout zeroes random units in training
  • Data augmentation varies the inputs

Stop early

  • Watch validation loss during training
  • Keep the checkpoint where it bottoms out
Notice how every technique limits how closely the model can fit noise in the training data, trading a bit of fit for better generalization.

Regularization is anything you do to a learning algorithm to make it generalize better, even at some cost to how well it fits the training data. The standard deep learning textbook defines it as a change meant to reduce generalization error “but not its training error”. It is the main set of tools against overfitting.

The intuition is a preference for simpler explanations. A flexible model can fit its training examples in many ways, most of which pick up noise. Regularization tilts training toward the smoother, simpler fits that are more likely to hold up on new data.

Three ways to do it

The diagram groups the common techniques by how they work.

Penalize large weights. Add a term to the loss that grows with the size of the weights, scaled by a strength setting, λ. L2 regularization adds the sum of squared weights, which shrinks all weights smoothly toward zero. L1 regularization adds the sum of absolute values, which tends to push many weights to exactly zero, giving a sparse model that effectively selects features. Weight decay shrinks the weights a little at every update. For plain SGD it’s equivalent to L2, but not for adaptive optimizers like Adam, which is why AdamW (2017) applies the decay separately.

Inject noise. Dropout switches off a random subset of units at each training step, so the network can’t rely on any single unit and effectively trains many thinned sub-networks at once. Data augmentation varies the inputs, with flips, crops or colour shifts on images, so the model sees each example in many forms.

Stop early. Early stopping tracks loss on a held-out validation set and keeps the checkpoint where it was lowest, before the model starts memorizing.

Other methods count too, such as label smoothing, and so does simply using more data or a smaller model.

Tuning the strength

Regularization strength is a hyperparameter. Too little and the model overfits; too much and it underfits, unable to fit even the training data. The right value is found by trying several and comparing validation scores, which is the bias–variance trade-off in practice.

The catch

Classical theory says regularization is what stops big models from memorizing, but deep networks complicate that story. A 2016 study found that standard image networks could still fit completely random labels with dropout, weight decay and augmentation switched on, and that the same networks with every regularizer turned off still generalized well on real data. Explicit regularizers help, but much of what makes large networks generalize comes from the data, the architecture and the training process itself.

Where it sits

Before this

Overfitting
Regularization

Explore nearby

In the research

All papers →

A paper that builds on Regularization .

Sources

  1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; chapter 7, Regularization for Deep Learning
  2. Loshchilov and Hutter, "Decoupled Weight Decay Regularization" . 2017; ICLR 2019 (AdamW)
  3. Zhang et al., "Understanding deep learning requires rethinking generalization" . 2016; ICLR 2017