Landmark 5 stops to get here · leads to 1

Dropout

A regularization technique that randomly switches off units during training so the network can't lean on any one of them, which reduces overfitting.

Your route here

5 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  4. Training Data ✓ understood

    The examples a model learns its weights from, kept separate from the validation and test data used to check how well it generalizes.

  5. Overfitting ✓ understood

    When a model fits its training data too closely, noise included, so it scores well on examples it has seen and poorly on new ones.

  6. Dropout · you are here ✓ understood

Picture it

During training

  • Random neurons zeroed each step
  • Rate p, e.g. 0.1 to 0.5
  • Survivors scaled up by 1/(1−p)
  • No neuron can rely on another

At inference

  • Every neuron is active
  • No rescaling needed
  • Deterministic output
  • Approximates the ensemble
Notice that dropout only happens during training; at inference the full network runs, averaging the thinned ones.

Dropout is a regularization technique for neural networks: during training, each unit’s output is randomly set to zero on every step, so the network has to work with a different, partly disabled version of itself each time. At inference nothing is dropped. It’s one of the cheapest defences against overfitting, usually a single line of code.

Hinton and colleagues introduced it in a 2012 paper, and the 2014 JMLR paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov gave the full treatment, reporting gains in vision, speech recognition, document classification and computational biology.

Why switching units off helps

In a large network trained on limited data, units can form fragile partnerships: one feature detector is only useful alongside a few specific others, a pattern that fits the training set and little else. The papers call this co-adaptation. If any partner might vanish on the next step, those partnerships stop paying off, and each unit is pushed toward features that are useful on their own.

The diagram’s two columns show a second way to see it. A network with n units contains 2^n possible “thinned” sub-networks, all sharing weights, and each training step samples and updates one of them. Training with dropout is like training a vast ensemble for the price of one model. At test time the full network, with its weights rescaled, approximates averaging all of those thinned networks’ predictions.

The rate, and the rescaling

The dropout rate is a hyperparameter, and the convention is easy to trip over. The 2014 paper uses p for the probability of keeping a unit, with typical values of 0.5 to 0.8 for hidden layers and 0.8 for real-valued inputs. PyTorch uses p for the probability of dropping one, with a default of 0.5.

Something has to compensate for the missing units, or activations at test time would be larger than anything seen in training. The original method multiplies the weights by the keep probability at test time. Frameworks such as PyTorch do the reverse: they scale the surviving activations up by 1/(1 − p) during training, so evaluation is a plain forward pass.

The catch

Dropout makes gradients noisy. The 2014 paper recommends compensating with a learning rate 10 to 100 times higher and high momentum, and warns that heavy dropout needs more units, which slows training. It also has to be switched off for evaluation: leaving a model in training mode makes its predictions change from run to run. Other regularizers, such as early stopping and weight decay, are often used alongside it.

Where it sits

Dropout

Leads to

Maxout

Explore nearby

In the research

All papers →

2 papers that build on Dropout .

Sources

  1. Hinton et al., "Improving neural networks by preventing co-adaptation of feature detectors" . 2012
  2. Srivastava et al., "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" . Journal of Machine Learning Research 15, 2014; sections 1 and A.2–A.4
  3. PyTorch documentation, torch.nn.Dropout . Drop-probability convention and 1/(1 − p) training-time scaling