Landmark 4 stops to get here · leads to 4

Learning Rate

The hyperparameter that sets how big a step gradient descent takes on each update; too high makes training unstable, too low makes it crawl.

Your route here

4 stops · basics first
  1. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  2. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  3. Loss Function ✓ understood

    A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

  4. Gradient Descent ✓ understood

    An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

  5. Learning Rate · you are here ✓ understood

Picture it

stepslossdiverges ↑too small (η = 0.02)good (η = 0.3)too large (η = 1.05)
Notice the same problem three ways: too small a rate crawls, a good rate drops fast, and too large a rate makes loss blow up.

The learning rate is the number that sets how big a step gradient descent takes each time it updates a model’s weights. The gradient says which direction reduces the loss; the learning rate says how far to move in that direction. It is a hyperparameter: you choose it, the model doesn’t learn it.

It is usually the single most consequential setting in training. The Deep Learning textbook puts it plainly: if you have time to tune only one hyperparameter, tune the learning rate.

Too big, too small

The diagram runs the same problem at three rates, and the pattern is general.

  • Too small: every step is safe but tiny. Training crawls, and it can stall on a plateau with the loss still high.
  • About right: the loss drops quickly and settles.
  • Too large: steps overshoot the valley floor and land higher up the other side. The loss bounces around or blows up. For an idealized quadratic loss, any rate at least twice the optimal one makes the error go up rather than down.

Good values span several orders of magnitude, so they are searched on a log scale. The textbook’s example search samples rates between 0.1 and 0.00001.

Changing it over time

A single fixed rate is rarely best for a whole run, so most training uses a learning rate schedule.

  • Warmup starts small and ramps up over the first part of training, which avoids unstable early updates. A 2017 Facebook study used a five-epoch warmup to train an ImageNet model with batches of 8,192 images.
  • Decay lowers the rate as training goes on. With stochastic gradient descent, each step uses a noisy estimate of the gradient, and that noise doesn’t vanish near a minimum. A shrinking step lets the model settle instead of jittering around.
  • Cosine annealing, from 2016 work on warm restarts, lowers the rate along a smooth half-cosine curve instead of in steps.

The catch

The right learning rate depends on everything else. The same Facebook study found that when you multiply the batch size by k, multiplying the learning rate by k keeps accuracy steady, up to a limit. Change the optimizer, the model size or the batch size and a rate that worked before may diverge or crawl. Adaptive optimizers like Adam adjust the step for each parameter, which can make training less sensitive to the choice, but they still have a learning rate, and it still needs tuning.

Where it sits

Explore nearby

In the research

All papers →

A paper that builds on Learning Rate .

Sources

  1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; section 8.3.1, Stochastic Gradient Descent; section 11.4, Selecting Hyperparameters
  2. Goyal et al., "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour" . 2017
  3. Loshchilov and Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts" . 2016; ICLR 2017