AI & ML / Deep learning / 08_backpropagation.md

Backpropagation

Updated 6 interview angles 4 min read source
On this page8
  1. Why it exists
  2. The chain rule, applied to a graph
  3. What it looks like by hand
  4. Autograd does this for you
  5. Where the gradient dies or explodes
  6. The part that is not backpropagation
  7. Related
  8. Interview angle

Backpropagation

The bridge between “I can fit a linear model” and “I can train a network”. It is the chain rule applied to a computation graph, run once backwards, and the reason deep learning is tractable at all.

Why it exists

A network with a million parameters needs a million partial derivatives per step. The naive approach — perturb each parameter, re-run the forward pass, see what changed — costs a million forward passes.

Backpropagation gets all of them in one backward pass, at roughly the cost of one forward pass. That is the entire contribution: not the derivatives, the reuse.

The chain rule, applied to a graph

text 
x ──[W₁]──▶ h ──[relu]──▶ a ──[W₂]──▶ ŷ ──[loss]──▶ L

forward:   compute and cache h, a, ŷ
backward:  ∂L/∂ŷ → ∂L/∂a → ∂L/∂h → ∂L/∂W₁

Each node knows two things: how to compute its output from its inputs, and how to turn a gradient on its output into a gradient on its inputs. Chain those backwards and every parameter gets its gradient.

The forward pass must cache its intermediates, because the backward pass needs them: ∂L/∂W₂ depends on a. That cache is why training uses far more memory than inference, and why torch.no_grad() makes evaluation cheaper.

What it looks like by hand

One linear layer, so the mechanics are visible:

python 
# forward
z = x @ W + b
y = relu(z)
loss = ((y - target) ** 2).mean()

# backward
dy = 2 * (y - target) / y.size          # ∂L/∂y
dz = dy * (z > 0)                       # relu' is 1 where z > 0
dW = x.T @ dz                           # ∂L/∂W
db = dz.sum(axis=0)
# passed to the previous layer
dx = dz @ W.T

dz = dy * (z > 0) is where ReLU’s gradient behaviour lives: it passes the gradient through unchanged where the unit was active and zeroes it where the unit was not. A unit that is never active gets no gradient and never recovers — the dying ReLU problem in Activation functions.

Autograd does this for you

python 
loss.backward()          # walks the graph, fills .grad on every leaf
optimizer.step()         # applies them
optimizer.zero_grad()    # or the next backward ADDS to these

PyTorch builds the graph as the forward pass runs — define-by-run — so control flow, loops and conditionals all differentiate correctly without a separate compilation step.

Gotcha: gradients accumulate. Forgetting zero_grad() sums this step’s gradients onto the last step’s, so the effective learning rate grows every iteration and training diverges for no visible reason. That accumulation is deliberate — it is how you simulate a large batch on small hardware, by running several backward passes before stepping.

Where the gradient dies or explodes

The chain rule multiplies. Across many layers, a factor consistently below 1 drives the product to zero and one above 1 drives it to infinity:

Problem Symptom Fix
Vanishing early layers stop learning residuals, ReLU, normalisation
Exploding loss becomes NaN gradient clipping, lower LR

Residual connections are the structural fix, and the reason is exactly this multiplication: y = x + f(x) gives the gradient an additive path straight through, so it reaches early layers without being multiplied down. That is why networks went from tens of layers to hundreds — see Normalisation and residual connections.

The part that is not backpropagation

Backprop computes gradients. It does not decide what to do with them — that is the optimiser, and conflating the two is a common slip:

python 
# backprop: what direction reduces the loss
loss.backward()

# optimiser: how far to move, and with what memory of past steps
optimizer.step()

SGD moves along the gradient. Adam keeps running estimates of the first and second moments and scales each parameter’s step by its own history, which is why it needs little tuning and why its state costs twice the model size in memory. See Training deep networks.

Interview angle 6

  • “What is backpropagation?” - the chain rule applied to the computation graph, evaluated once backwards. Its contribution is efficiency: every parameter’s gradient in one backward pass at roughly the cost of one forward pass, instead of one forward pass per parameter.
  • “Why does training use so much more memory than inference?” - the forward pass caches its intermediate activations, because the backward pass needs them to compute gradients. torch.no_grad() skips that, which is why evaluation is cheaper.
  • “What does zero_grad() do and why is it needed?” - gradients accumulate into .grad rather than replacing it. Forget it and each step sums onto the last, so the effective step size grows and training diverges. The accumulation is deliberate — it is how you fake a large batch on small hardware.
  • “Why do gradients vanish?” - the chain rule multiplies, so a factor consistently below one across many layers drives the product toward zero and early layers stop learning. Residual connections fix it structurally by giving the gradient an additive path that is not multiplied down.
  • “Is backpropagation the same as gradient descent?” - no. Backprop computes the gradients; the optimiser decides the step. SGD follows the gradient, Adam scales each parameter by running estimates of its first and second moments — which is why Adam’s state costs about twice the model size.
  • “How does ReLU affect the backward pass?” - it passes the gradient through where the unit was active and zeroes it where it was not. A unit that never activates receives no gradient and cannot recover, which is the dying ReLU problem.
Previous
Transfer learning