Standard 7 stops to get here · leads to 4

Adam Optimizer

An adaptive learning rate optimization algorithm combining momentum and RMSprop, widely used for training neural networks.

Your route here

7 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  3. Loss Function ✓ understood

    A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

  4. Gradient Descent ✓ understood

    An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

  5. Learning Rate ✓ understood

    The hyperparameter that sets how big a step gradient descent takes on each update; too high makes training unstable, too low makes it crawl.

  6. Optimizer ✓ understood

    The algorithm that turns gradients into weight updates during training, deciding how far and in which direction each parameter moves at every step.

  7. Momentum ✓ understood

    An optimization technique that accelerates gradient descent by accumulating past gradients, helping escape local minima.

  8. Adam Optimizer · you are here ✓ understood

Where it sits

Before this

Momentum Learning Rate Optimizer
Adam Optimizer

Leads to

AdamW LAMB Optimizer Lookahead Optimizer ZeRO

Explore nearby

Training AdamW Adam with decoupled weight decay, providing better regularization and often superior performance. Training RMSprop An optimizer using moving average of squared gradients to adapt learning rates, addressing AdaGrad's diminishing rates. Training Stochastic Gradient Descent A variant of gradient descent that updates parameters using gradients computed on a single random training example at a time (though often used to refer to mini-batch gradient descent). Training AdaGrad An optimizer that adapts learning rates for each parameter based on historical gradients, useful for sparse data. Training LAMB Optimizer Layer-wise Adaptive Moments optimizer for Batch training - enables very large batch training for transformers.

In the research

All papers →

A paper that builds on Adam Optimizer .