The learning rate is the number that sets how big a step gradient descent takes each time it updates a model’s weights. The gradient says which direction reduces the loss; the learning rate says how far to move in that direction. It is a hyperparameter: you choose it, the model doesn’t learn it.
It is usually the single most consequential setting in training. The Deep Learning textbook puts it plainly: if you have time to tune only one hyperparameter, tune the learning rate.
Too big, too small
The diagram runs the same problem at three rates, and the pattern is general.
- Too small: every step is safe but tiny. Training crawls, and it can stall on a plateau with the loss still high.
- About right: the loss drops quickly and settles.
- Too large: steps overshoot the valley floor and land higher up the other side. The loss bounces around or blows up. For an idealized quadratic loss, any rate at least twice the optimal one makes the error go up rather than down.
Good values span several orders of magnitude, so they are searched on a log scale. The textbook’s example search samples rates between 0.1 and 0.00001.
Changing it over time
A single fixed rate is rarely best for a whole run, so most training uses a learning rate schedule.
- Warmup starts small and ramps up over the first part of training, which avoids unstable early updates. A 2017 Facebook study used a five-epoch warmup to train an ImageNet model with batches of 8,192 images.
- Decay lowers the rate as training goes on. With stochastic gradient descent, each step uses a noisy estimate of the gradient, and that noise doesn’t vanish near a minimum. A shrinking step lets the model settle instead of jittering around.
- Cosine annealing, from 2016 work on warm restarts, lowers the rate along a smooth half-cosine curve instead of in steps.
The catch
The right learning rate depends on everything else. The same Facebook study found that when you multiply the batch size by k, multiplying the learning rate by k keeps accuracy steady, up to a limit. Change the optimizer, the model size or the batch size and a rate that worked before may diverge or crawl. Adaptive optimizers like Adam adjust the step for each parameter, which can make training less sensitive to the choice, but they still have a learning rate, and it still needs tuning.