Dropout is a regularization technique for neural networks: during training, each unit’s output is randomly set to zero on every step, so the network has to work with a different, partly disabled version of itself each time. At inference nothing is dropped. It’s one of the cheapest defences against overfitting, usually a single line of code.
Hinton and colleagues introduced it in a 2012 paper, and the 2014 JMLR paper by Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov gave the full treatment, reporting gains in vision, speech recognition, document classification and computational biology.
Why switching units off helps
In a large network trained on limited data, units can form fragile partnerships: one feature detector is only useful alongside a few specific others, a pattern that fits the training set and little else. The papers call this co-adaptation. If any partner might vanish on the next step, those partnerships stop paying off, and each unit is pushed toward features that are useful on their own.
The diagram’s two columns show a second way to see it. A network with n units contains 2^n possible “thinned” sub-networks, all sharing weights, and each training step samples and updates one of them. Training with dropout is like training a vast ensemble for the price of one model. At test time the full network, with its weights rescaled, approximates averaging all of those thinned networks’ predictions.
The rate, and the rescaling
The dropout rate is a hyperparameter, and the convention is easy to trip over. The 2014 paper uses p for the probability of keeping a unit, with typical values of 0.5 to 0.8 for hidden layers and 0.8 for real-valued inputs. PyTorch uses p for the probability of dropping one, with a default of 0.5.
Something has to compensate for the missing units, or activations at test time would be larger than anything seen in training. The original method multiplies the weights by the keep probability at test time. Frameworks such as PyTorch do the reverse: they scale the surviving activations up by 1/(1 − p) during training, so evaluation is a plain forward pass.
The catch
Dropout makes gradients noisy. The 2014 paper recommends compensating with a learning rate 10 to 100 times higher and high momentum, and warns that heavy dropout needs more units, which slows training. It also has to be switched off for evaluation: leaving a model in training mode makes its predictions change from run to run. Other regularizers, such as early stopping and weight decay, are often used alongside it.