Landmark 5 stops to get here · leads to 1

Dropout

A regularization technique that randomly deactivates neurons during training to prevent overfitting and improve generalization.

Your route here

5 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.

  3. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  4. Training Data ✓ understood

    The examples a model learns its weights from, kept separate from the validation and test data used to check how well it generalizes.

  5. Overfitting ✓ understood

    When a model fits its training data too closely, noise included, so it scores well on examples it has seen and poorly on new ones.

  6. Dropout · you are here ✓ understood

Picture it

During training

  • Random neurons zeroed each step
  • Rate p, e.g. 0.1 to 0.5
  • No neuron can rely on another
  • Acts like many thinned networks

At inference

  • Every neuron is active
  • Scaled to match training
  • Deterministic output
  • Approximates the ensemble
Notice that dropout only happens during training; at inference the full network runs, averaging the thinned ones.

Where it sits

Before this

Neural Network Overfitting
Dropout

Leads to

Maxout

Explore nearby

Training Regularization Techniques to prevent overfitting by adding constraints or penalties to the model (L1, L2, dropout, early stopping). Foundations Ensemble Learning Combining multiple models to produce better predictions than any individual model (bagging, boosting, stacking). Neural Networks Batch Normalization A technique that normalizes layer inputs to stabilize and accelerate training by reducing internal covariate shift. Training Early Stopping Stopping training when validation performance stops improving, preventing overfitting. Training Weight Decay A regularization technique that shrinks weights toward zero during optimization. Equivalent to L2 regularization in standard SGD, but differs when using adaptive optimizers like Adam.

In the research

All papers →

2 papers that build on Dropout .