Backpropagation: how a neural network learns

The problem: one number and billions of knobs

Picture a network as a machine with billions of knobs. You feed it an input, it produces an output, and comparing that output with the right answer gives one number, the loss: zero means perfect, larger means worse. Your job is to turn the knobs so that the number goes down, for this input and for all the others.

The obvious method is to try each knob in turn: turn it a little, run the input through again, see whether the loss went up or down, put it back, move on. This works, and it is hopeless at scale. A model with 7 billion weights would need 7 billion forward passes to learn which way to turn each knob once.

Trial and error:
  for each weight: nudge it, rerun the network, compare
  cost = N forward passes  (N = number of weights)
  N = 7,000,000,000  ->  7 billion forward passes per update

Backpropagation:
  1 forward pass + 1 backward pass
  cost = about 3 forward passes, whatever N is

Backpropagation gets the answer for all N weights in one pass. The trick is that the network is not a bag of independent knobs but a chain of simple operations, and each weight affects the loss only through what lies downstream of it. Working backward from the loss, you can measure the loss's sensitivity to each intermediate value once and reuse it for everything upstream. The rest of this guide is that idea, spelled out.

The loss function

A loss function turns a prediction and a correct answer into a single number that measures wrongness. It should be smallest when the prediction is right, grow as the prediction gets further from right, and be smooth, so that a tiny change to the prediction produces a tiny, predictable change in the loss. Smoothness is what makes slopes exist, and slopes are what backpropagation computes.

For the toy network the loss is squared error, L = (y - t)^2, where y is the output and t is the target. Squaring makes the loss positive whether the output is too high or too low, and makes big misses cost far more than small ones. The companion guide's network outputs y = -0.5 on the input x1 = 2, x2 = 1; set the target to t = 1.

output   y = -0.5
target   t = 1
L = (y - t)^2 = (-0.5 - 1)^2 = (-1.5)^2 = 2.25

A language model does not output a single number. At each position in the text it outputs a probability for every token in its vocabulary, its guess at what comes next. The loss at that position is the probability it gave to the token that actually came next, run through a logarithm and flipped to be positive. This is called cross-entropy: a probability of 1 for the right token gives a loss of 0, and the loss climbs fast as that probability shrinks. The loss for a batch of text is the average over every position, and training a model means driving that average down. Backpropagation needs only one thing from whichever loss you pick: its slope with respect to the output, which for squared error is 2(y - t).

probability given to the correct next token  ->  loss
  1.00  ->  0
  0.50  ->  0.69
  0.10  ->  2.30
  0.01  ->  4.61
(loss = minus the natural log of the probability)

Gradient descent with one weight

Start with a loss that depends on one weight alone. A derivative is how much an output changes per tiny change in an input; on a graph, it is the slope of the curve. The gradient is the same idea for a function of many inputs: the list of slopes, one per weight. With one weight, the gradient is just the slope.

The update rule of gradient descent is one line: w = w - lr * gradient. If the slope is positive, raising w raises the loss, so lower w. If the slope is negative, raise w. Either way the weight moves against the slope, and the learning rate lr scales how far it moves. The rule does not know where the bottom of the curve is, only which way is downhill from where it stands.

loss L = (w - 3)^2     slope = 2(w - 3)     lr = 0.25

w = 0      L = 9       slope = -6    w -> 0 - 0.25*(-6) = 1.5
w = 1.5    L = 2.25    slope = -3    w -> 1.5 + 0.75 = 2.25
w = 2.25   L = 0.5625  slope = -1.5  w -> 2.25 + 0.375 = 2.625
w = 2.625  L = 0.1406

The steps shrink on their own: the learning rate never changed, but the slope flattens near the bottom, so each step is smaller than the last. The flip side is that a step size that is fine near the bottom can be far too big high on the slope, as the toy network is about to show.

The chain rule in plain words

A weight in the first layer of the toy network does not touch the loss directly. It changes the sum inside a hidden neuron, which changes that neuron's output, which changes the network's output, which changes the loss. To know how the loss responds to that weight, you need the derivative along the whole chain.

The chain rule says the derivative along a chain is the product of the derivatives of each link. If a small nudge to w moves u by twice as much, and a small nudge to u moves y by three times as much, then a nudge to w moves y by six times as much.

u = 2w    (nudging w by 0.1 moves u by 0.2)    du/dw = 2
y = 3u    (nudging u by 0.2 moves y by 0.6)    dy/du = 3
dy/dw = dy/du * du/dw = 3 * 2 = 6
check: w = 1 -> u = 2 -> y = 6;  w = 1.1 -> u = 2.2 -> y = 6.6

Two more facts complete the toolkit. First, if a value feeds several downstream paths, its total effect on the loss is the sum of its effects along each path. The toy network needs no adding, since each hidden neuron feeds only the output, but in a wide network every hidden value fans out to many neurons. Second, each link has a simple derivative. For a product w * x, the derivative with respect to x is w and with respect to w is x. Adding a bias has derivative 1. ReLU, which outputs its input when positive and 0 otherwise, has derivative 1 when its input was positive and 0 when it was not: it either passes the gradient through untouched or blocks it.

Backpropagation itself

Backpropagation is the chain rule applied to a whole network in a specific, economical order. It has two halves.

The forward pass runs the input through the network as usual but keeps every intermediate value: each weighted sum, each activation, and the output. The derivative at each link depends on what flowed through it: the ReLU's derivative depends on whether its input was positive, and the derivative of v1 * h1 with respect to v1 is h1, so h1 must still be around.

The backward pass starts at the loss and walks toward the input. First it computes how the loss changes with the output, dL/dy. From that it gets two things for the output layer: the gradient for each weight (dL/dy times the input that weight touched) and the gradient for each input (dL/dy times the weight that input touched). The second set, dL/dh1 and dL/dh2, is exactly what the hidden layer needs to repeat the same two moves, and so on down to the input.

The economy is in the reuse: dL/dy is computed once and used for v1, v2, c, h1, and h2; dL/dh1 is computed once and used for w1, w2, and b1. The backward pass does about two multiplications per weight, so it costs roughly twice the forward pass whether the network has nine weights or nine billion, where trial and error cost one forward pass per weight. This is why training deep networks is possible at all.

One training step, in full

Here is the whole step. The companion guide covers the forward pass in detail, so it is shown briefly.

Forward pass

inputs    x1 = 2,  x2 = 1
hidden 1  w1 = 0.5, w2 = -1,  b1 = 1
hidden 2  w3 = 1,   w4 = 0.5, b2 = -0.5
output    v1 = 1,   v2 = -1,  c = 0.5

z1 = 0.5*2 + (-1)*1 + 1    = 1     h1 = ReLU(1) = 1
z2 = 1*2 + 0.5*1 + (-0.5)  = 2     h2 = ReLU(2) = 2
y  = 1*1 + (-1)*2 + 0.5    = -0.5
L  = (y - t)^2 = (-0.5 - 1)^2 = 2.25    target t = 1

Both hidden neurons received a positive sum, so both ReLUs are active, each has derivative 1, and the gradient will pass through both unchanged.

Backward pass, output layer

The slope of squared error is twice the difference between output and target. Then each output weight gets that slope times its input, and each hidden activation gets that slope times its weight.

dL/dy  = 2(y - t) = 2(-0.5 - 1) = -3

dL/dv1 = dL/dy * h1 = -3 * 1 = -3
dL/dv2 = dL/dy * h2 = -3 * 2 = -6
dL/dc  = dL/dy * 1  = -3

dL/dh1 = dL/dy * v1 = -3 * 1    = -3
dL/dh2 = dL/dy * v2 = -3 * (-1) = 3

Read the signs. dL/dv2 = -6 says that raising v2 would lower the loss, and fast: v2 is negative and multiplies the largest hidden activation, so it is doing most of the work of dragging y below the target.

Backward pass, hidden layer

Each hidden neuron now has the gradient of its output. The ReLU multiplies it by 1, since both were active, and then the same two moves as before: each weight gets the gradient times its input, and the bias gets it times 1.

through ReLU:  dL/dz1 = dL/dh1 * 1 = -3
               dL/dz2 = dL/dh2 * 1 = 3

dL/dw1 = dL/dz1 * x1 = -3 * 2 = -6
dL/dw2 = dL/dz1 * x2 = -3 * 1 = -3
dL/db1 = dL/dz1      = -3

dL/dw3 = dL/dz2 * x1 = 3 * 2 = 6
dL/dw4 = dL/dz2 * x2 = 3 * 1 = 3
dL/db2 = dL/dz2      = 3

That is the full gradient: nine numbers, one per weight, from one backward pass, with the inputs x1 and x2 saved from the forward pass needed at the last step.

The update

Gradient descent now moves every weight against its gradient, scaled by the learning rate: with lr = 0.05, a gradient of -6 moves a weight up by 0.3.

v1 = 1    - 0.05*(-3) = 1.15
v2 = -1   - 0.05*(-6) = -0.7
c  = 0.5  - 0.05*(-3) = 0.65
w1 = 0.5  - 0.05*(-6) = 0.8
w2 = -1   - 0.05*(-3) = -0.85
b1 = 1    - 0.05*(-3) = 1.15
w3 = 1    - 0.05*(6)  = 0.7
w4 = 0.5  - 0.05*(3)  = 0.35
b2 = -0.5 - 0.05*(3)  = -0.65

Forward pass again

Run the same input through the updated network.

z1 = 0.8*2 + (-0.85)*1 + 1.15 = 1.6 - 0.85 + 1.15 = 1.9
h1 = ReLU(1.9) = 1.9
z2 = 0.7*2 + 0.35*1 + (-0.65) = 1.4 + 0.35 - 0.65 = 1.1
h2 = ReLU(1.1) = 1.1
y  = 1.15*1.9 + (-0.7)*1.1 + 0.65
   = 2.185 - 0.77 + 0.65 = 2.065
L  = (2.065 - 1)^2 = 1.134

The loss fell from 2.25 to about 1.13. The direction of every change was right. But look at the output: it went from -0.5 to 2.065, straight past the target of 1. The step overshot.

Why the step overshot

The gradient is a local measurement: which way is downhill right here, and how steep, exact only for an infinitely small step. A learning rate of 0.05 took a much bigger step than that, and all nine weights moved at once, so their effects compound. v1 went up, and h1 (which v1 multiplies) also went up, so the product v1 * h1 rose more than either change alone would predict, from 1 to 2.185. The same happened, in the other direction, for v2 * h2.

A smaller learning rate steps more gently, and a larger one can make things worse. Here is the same gradient with three different rates.

lr = 0.01   y = 0.035   L = 0.93   (short of target, lowest loss)
lr = 0.05   y = 2.065   L = 1.13   (past target, loss still down)
lr = 0.20   y = 8.46    L = 55.65  (h2 shut off, loss exploded)

With lr = 0.2 the second hidden neuron's sum went negative, its ReLU switched off, and the output ran away: divergence in miniature. There is no formula for the right learning rate; it is tuned by trial and changed over the run. No single step is meant to land on the target anyway: the next step uses a different example, with its own gradient, and the weights have to serve all of them at once.

Doing it at scale

Everything above was one example, one gradient, one step. A real training run differs in four ways.

Minibatches

The loss that matters is the average over the entire dataset, and computing its exact gradient would mean a forward and backward pass over every example before a single update, which is far too slow. Instead, each step takes a minibatch, a random sample of examples, averages their gradients, and updates on that. The estimate is noisy but cheap, and the noise turns out to be mostly harmless and sometimes helpful. This is stochastic gradient descent; stochastic means the gradient is a noisy estimate. For language models a batch is measured in tokens, and a few million per step is common.

Learning-rate schedules

The learning rate is rarely a constant. Most large runs start it near zero and raise it over the first few thousand steps, a warmup, because freshly initialized weights produce large, erratic gradients and a full-size step early on can wreck the run. After warmup the rate decays, often along a cosine curve, so that late steps are small and the weights settle rather than bounce around the bottom of the valley.

Adam

Plain gradient descent uses one learning rate for every weight. Adam, the optimizer behind nearly every large language model (usually as its AdamW variant), gives each weight its own effective step size. It keeps two running averages per weight: the average gradient, which smooths noise and adds momentum, and the average squared gradient, which tracks how big that weight's gradients tend to be. The step is the first divided by the square root of the second, so a weight with consistently large gradients takes smaller steps and one with tiny gradients takes larger ones. It costs two extra numbers per weight and makes training far less sensitive to the learning rate.

Epochs and the scale of a real run

One pass through the entire dataset is an epoch. Small datasets are cycled through many times; language model pre-training usually sees its data once or a few times. Pre-training a large model means running the loop in this guide on trillions of tokens across thousands of GPUs for weeks to months. Each GPU computes the gradient for its slice of the batch, the gradients are averaged across all of them, and the same update is applied everywhere so every copy of the weights stays identical. The algorithm is the one that updated nine weights above.

common estimate: 6 * parameters * tokens operations
  (2 per parameter per token forward, 4 backward)
model  7,000,000,000 parameters
data   2,000,000,000,000 tokens
6 * 7e9 * 2e12 = 8.4e22 operations

reported: Llama 2 7B, 2 trillion tokens, ~184,000 GPU-hours
  = ~7,700 GPU-days: 1,000 GPUs for about 8 days
implied: 8.4e22 / (184,000 * 3600 s) = 1.3e14 ops/s per GPU

Things that go wrong

The algorithm is simple. Keeping it working on a deep network with billions of weights is not.

Vanishing and exploding gradients

The gradient for a weight near the input is a product with one factor per layer between it and the loss. If those factors tend to be smaller than 1, the product shrinks toward zero and the early layers barely move: the gradient vanishes. If they tend to be larger than 1, it blows up and the update wrecks the weights: the gradient explodes.

50 layers, each contributing a factor of 0.5:
  0.5^50 = 0.00000000000000089  (about 9e-16)
50 layers, each contributing a factor of 2:
  2^50 = 1,125,899,906,842,624  (about 1.1e15)

Two design choices exist largely to fight this. ReLU-type activations have derivative 1 wherever they are active, so they do not shrink the gradient the way older sigmoid activations did (a sigmoid's derivative is at most 0.25). Residual connections add each layer's input straight to its output, which gives the gradient a path through the layer with derivative exactly 1. That path is why transformers can be stacked a hundred layers deep; the transformer guide shows where those connections sit. Normalization, careful initialization, and clipping oversized gradients do the rest.

The toy example showed a milder failure: with lr = 0.2 the second neuron's ReLU switched off. If its sum stayed negative for every training example, its gradient would be 0 forever, a dead ReLU that has stopped learning for good. Variants such as leaky ReLU keep a small slope on the negative side so that a neuron can recover.

Overfitting

Driving the training loss down is not the goal; low loss on data the network has never seen is. A network with enough weights can memorize its training set while doing badly on new data. That is overfitting, and it shows up as a training loss that keeps falling while the loss on held-out data stops falling and starts to rise.

More data is the strongest remedy, because there is less room to memorize when there is more to fit. Regularization covers the techniques that make memorizing harder: weight decay pulls every weight gently toward zero each step, dropout randomly zeroes activations during training so no single path can be relied on, and early stopping ends the run when held-out loss stops improving. Pre-training, which sees each token once or a few times, keeps training and held-out losses close, though models still memorize sequences that recur often; fine-tuning on a small dataset brings the problem back quickly.

Local minima and saddle points

On the one-weight bowl, the slope is zero only at the bottom. A bumpier curve can have dips where the slope is zero but the loss is not at its lowest, a local minimum, and gradient descent would stop there. It is natural to worry that a billion-dimensional surface is full of such traps.

The evidence so far suggests the worry is mostly misplaced. A local minimum has to curve upward in every one of those billion directions at once. Researchers have found that flat points in high-dimensional loss surfaces are overwhelmingly saddle points instead, curving up in some directions and down in others, and that the local minima that do exist tend to have loss close to the global best (Dauphin and colleagues, 2014). A saddle point slows training rather than stopping it, since there is always a downhill direction, and minibatch noise plus momentum tend to push the weights off it. That is an empirical picture, not a theorem, and runs do still stall or diverge for reasons that take real work to diagnose.

Automatic differentiation

Nobody derives the gradients of a real network by hand. A framework such as PyTorch records every operation as the forward pass runs, building a graph of what was computed from what. Each operation (multiplication, addition, ReLU, matrix multiply, softmax) knows its own local derivative. When you ask for the gradient of the loss, the framework walks that graph in reverse, applying the chain rule at every node, and deposits a gradient on every weight. This is automatic differentiation, and backpropagation is its reverse-mode variant applied to a neural network.

Reverse mode fits because a network has a huge number of inputs (the weights) and one output (the loss), and one backward sweep gives the derivative of that output with respect to all of them. The alternative, forward mode, handles one input per sweep, which would mean one sweep per weight, the trial-and-error cost from the start of this guide. The whole training loop in a framework is a few lines.

y = model(x)               # forward pass, graph recorded
loss = (y - t) * (y - t)   # squared error
loss.backward()            # backprop: fills w.grad for every w
optimizer.step()           # gradient descent: w -= lr * w.grad
optimizer.zero_grad()      # clear gradients for the next step

Because the differentiation is automatic, an architecture can be changed freely and the gradients follow. The one requirement is that every operation must have a usable derivative almost everywhere, which is why a hard step such as "pick the biggest" is usually replaced by a soft one such as softmax.

History, and the brain

Seppo Linnainmaa described reverse-mode differentiation in a 1970 thesis, and related ideas had appeared in control theory in the early 1960s. Paul Werbos described the method in his 1974 thesis and later spelled out its use for training multilayer networks. It spread only after Rumelhart, Hinton, and Williams published a short paper in 1986 showing that networks trained this way learned useful internal representations in their hidden layers, which was the point in doubt at the time. The Wikipedia article has the full lineage.

Whether the brain does anything like this is an open question, and the honest answer leans toward no, at least not in this form. Backpropagation needs a separate backward phase, an exact error signal delivered to every connection, and a backward path that mirrors the forward weights precisely, and no known biological mechanism provides those. Some researchers study approximations a brain might plausibly implement, but most neuroscientists doubt that real neurons run this algorithm. The resemblance between artificial and biological neurons is loose, and between backpropagation and whatever synapses actually do, looser still.

What this means for the model you chat with

Training ends, the weights are saved, and from then on the model runs forward passes only. When you send a message, the text goes through the network, a probability distribution comes out, a token is picked, and the loop repeats. There is no target, no loss, no backward pass, and no update. The weights that produced your answer are, to the last digit, the weights saved when training stopped.

The model still appears to learn within a conversation. You correct it, and its next answer reflects the correction. That is not learning in the sense of this guide. Your earlier messages sit in the context window, and the next forward pass reads them as input. Close the conversation and nothing remains. A fact the model was not trained on is usable exactly as long as it stays in the window.

Getting new knowledge into the weights themselves means running this algorithm again: further pre-training on new data, or fine-tuning on a smaller set. Either is a training job with a loss, a backward pass, and a learning rate, not something a conversation can trigger. This is why every model has a knowledge cutoff, and why answers about anything after it come from the context, from retrieval, or from nowhere.