Standard 4 stops to get here · leads to 4

Optimizer

The algorithm that turns gradients into weight updates during training, deciding how far and in which direction each parameter moves at every step.

Your route here

4 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  3. Loss Function ✓ understood

    A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

  4. Gradient Descent ✓ understood

    An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

  5. Optimizer · you are here ✓ understood

Picture it

Forward passRun a batch, measure the lossBackward passBackprop computes the gradientsOptimizer stepTurn gradients into weight updatesUpdate stateMomentum and running averages…One training step
  1. 01 Forward pass Run a batch, measure the loss
  2. 02 Backward pass Backprop computes the gradients
  3. 03 Optimizer step Turn gradients into weight updates
  4. 04 Update state Momentum and running averages carry over

↺ back to 01 · One training step

The gradient says which way is downhill; the optimizer decides how far to move each weight, and remembers past steps to do it better.

An optimizer is the part of training that actually changes the model. After each batch, backpropagation works out the gradient: how the loss would change if each weight moved a little. The optimizer takes those gradients and decides the update. Every optimizer is a variation on gradient descent, stepping each weight a little way downhill, but they differ in how they size and steer those steps.

The family

  • SGD: move each weight against its gradient, scaled by one global learning rate. Simple, and still competitive when well tuned.
  • Momentum: keep a running average of past gradients, so steps build speed in consistent directions and damp out zig-zags.
  • Adaptive methods such as AdaGrad, RMSProp and Adam: track statistics of each weight’s gradients and give each weight its own effective step size. Adam combines momentum with per-weight scaling, and it and its close variant AdamW are the usual defaults for training large neural networks.

What it costs

Optimizers are not free. Adam keeps two extra numbers for every parameter, running averages of its gradient and of its squared gradient, so its state takes about twice the memory of the weights themselves. For very large models that overhead shapes how training is split across machines.

The catch

No optimizer is best everywhere. The same model can converge, stall or diverge depending on the optimizer, its learning rate and the schedule that changes that rate over time. In practice the three are chosen together and tuned as a set, which is why published training recipes spell out all of them.

Where it sits

Before this

Gradient Descent Loss Function
Optimizer

Leads to

Adam Optimizer Momentum Stochastic Gradient Descent RMSprop

Explore nearby

Training Stochastic Gradient Descent A variant of gradient descent that updates parameters using gradients computed on a single random training example at a time (though often used to refer to mini-batch gradient descent). Training Adam Optimizer An adaptive learning rate optimization algorithm combining momentum and RMSprop, widely used for training neural networks. Training Momentum An optimization technique that accelerates gradient descent by accumulating past gradients, helping escape local minima. Training Learning Rate The hyperparameter that sets how big a step gradient descent takes on each update; too high makes training unstable, too low makes it crawl. Training RMSprop An optimizer using moving average of squared gradients to adapt learning rates, addressing AdaGrad's diminishing rates. Training Learning Rate Schedule A strategy for adjusting the learning rate during training (decay, warm-up, cosine annealing) to improve convergence.

Sources

  1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; chapter 8, Optimization for Training Deep Models
  2. Sebastian Ruder, "An overview of gradient descent optimization algorithms" . arXiv, 2016
  3. Diederik P. Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization" . ICLR 2015