Backpropagation is the algorithm that works out, for every weight in a neural network, how much a small change to that weight would change the loss. Those numbers, together called the gradient, are exactly what gradient descent needs to know which way to nudge each weight. Backpropagation computes the gradient; a separate optimizer uses it to learn.
The problem it solves is scale. A network can have billions of weights. Testing each one separately, wiggling it and rerunning the network, would take billions of runs per training step. Backpropagation gets all of them in a single backward sweep that costs a small constant multiple of one forward pass, typically two to three times.
How it works
The diagram walks through one step. The forward pass runs an input through the layers to a prediction and a loss, and keeps each layer’s intermediate values. Then the backward pass starts at the loss and moves toward the input.
The engine is the chain rule from calculus. Each layer only needs two local facts: how its output changes with its inputs, and how it changes with its own weights. Given the gradient arriving from the layer above, it multiplies by those local derivatives to get the gradient for its weights and the gradient to pass down to the layer below. Each layer reuses the work already done above it, which is why the whole sweep is so cheap.
Modern frameworks do this automatically. You write the forward computation, and the library records it and runs the backward pass for you. That general technique is called reverse-mode automatic differentiation, and backpropagation is its best-known special case.
A short history
The underlying method was worked out more than once, in control theory and numerical computing, from the 1960s and 1970s. It became the standard way to train neural networks after Rumelhart, Hinton and Williams’ 1986 Nature paper, which showed that it lets hidden units learn useful internal features on their own.
The catch
Memory is the price. The backward pass needs the intermediate values from the forward pass, so every layer’s activations must be stored until the gradient reaches them. That storage grows with the size of the computation, not just the number of weights, so long inputs and big batches get expensive fast. Gradient checkpointing trades compute back for memory by keeping only some activations and recomputing the rest.
The chain rule also multiplies many factors together, and in a deep network they can shrink or grow at every layer. That produces vanishing or exploding gradients, which careful initialization, normalization layers and residual connections are designed to counter.