Landmark 6 stops to get here · leads to 5

Backpropagation

The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent.

Your route here

6 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  4. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  5. Loss Function ✓ understood

    A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

  6. Gradient Descent ✓ understood

    An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

  7. Backpropagation · you are here ✓ understood

Picture it

  1. 01 Forward pass Inputs flow through to a prediction
  2. 02 Compute loss Compare prediction to the target
  3. 03 Gradient at output How the loss changes with the output
  4. 04 Propagate backward Chain rule, layer by layer
  5. 05 Gradient per weight Handed to gradient descent
Notice how the error signal travels backward, reusing each layer's gradient to compute the one before it.

Backpropagation is the algorithm that works out, for every weight in a neural network, how much a small change to that weight would change the loss. Those numbers, together called the gradient, are exactly what gradient descent needs to know which way to nudge each weight. Backpropagation computes the gradient; a separate optimizer uses it to learn.

The problem it solves is scale. A network can have billions of weights. Testing each one separately, wiggling it and rerunning the network, would take billions of runs per training step. Backpropagation gets all of them in a single backward sweep that costs a small constant multiple of one forward pass, typically two to three times.

How it works

The diagram walks through one step. The forward pass runs an input through the layers to a prediction and a loss, and keeps each layer’s intermediate values. Then the backward pass starts at the loss and moves toward the input.

The engine is the chain rule from calculus. Each layer only needs two local facts: how its output changes with its inputs, and how it changes with its own weights. Given the gradient arriving from the layer above, it multiplies by those local derivatives to get the gradient for its weights and the gradient to pass down to the layer below. Each layer reuses the work already done above it, which is why the whole sweep is so cheap.

Modern frameworks do this automatically. You write the forward computation, and the library records it and runs the backward pass for you. That general technique is called reverse-mode automatic differentiation, and backpropagation is its best-known special case.

A short history

The underlying method was worked out more than once, in control theory and numerical computing, from the 1960s and 1970s. It became the standard way to train neural networks after Rumelhart, Hinton and Williams’ 1986 Nature paper, which showed that it lets hidden units learn useful internal features on their own.

The catch

Memory is the price. The backward pass needs the intermediate values from the forward pass, so every layer’s activations must be stored until the gradient reaches them. That storage grows with the size of the computation, not just the number of weights, so long inputs and big batches get expensive fast. Gradient checkpointing trades compute back for memory by keeping only some activations and recomputing the rest.

The chain rule also multiplies many factors together, and in a deep network they can shrink or grow at every layer. That produces vanishing or exploding gradients, which careful initialization, normalization layers and residual connections are designed to counter.

Where it sits

Explore nearby

In the research

All papers →

2 papers that build on Backpropagation .

Sources

  1. Rumelhart, Hinton and Williams, "Learning representations by back-propagating errors" . Nature, 1986
  2. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning . MIT Press, 2016; section 6.5, Back-Propagation and Other Differentiation Algorithms
  3. Baydin et al., "Automatic differentiation in machine learning: a survey" . 2015; Journal of Machine Learning Research, 2018