Backpropagation

Backpropagation applies the chain rule from a neural network's output toward its input to compute the gradient of a loss with respect to every parameter.

Table of contents
- Squared-error backpropagation Backpropagation
- Sigmoid-softmax network gradients Backpropagation

Squared-error backpropagation

For squared loss

E = \sum_{k} (t_{k} - z_{k})^{2} /2

, let

e_{k} = (t_{k} - z_{k}) g_{k}^{'} (x_{k})

and

e_{j} = g_{j}^{'} (x_{j}) \sum_{k} w_{jk} e_{k}

. The chain rule gives

\partial E / \partial w_{jk} = - e_{k} z_{j}

and

\partial E / \partial w_{ij} = - e_{j} z_{i}

. A gradient descent step of size

η

Δ w_{jk} = η e_{k} z_{j}

and

Δ w_{ij} = η e_{j} z_{i}

, with every derivative evaluated at the same pre-update weights.

Sigmoid-softmax network gradients

For hidden activations

h_{j} = σ (b_{j} + \sum_{r} W_{j r} x_{r})

and class probabilities

p_{c} = softmax_{c} (a + Vh)

, a one-hot label

t

has log-likelihood

ℓ = \sum_{c} t_{c} lo g p_{c}

. Put

δ_{c} = t_{c} - p_{c}

and

Δ_{j} = h_{j} (1 - h_{j}) \sum_{c} V_{c j} δ_{c}

. Then

\partial_{a_{c}} ℓ = δ_{c}, \partial_{V_{c j}} ℓ = δ_{c} h_{j}, \partial_{b_{j}} ℓ = Δ_{j}, \partial_{W_{j r}} ℓ = Δ_{j} x_{r} .

These follow from the chain rule and give single-observation stochastic gradient descent updates for the negative log-likelihood.

 Ancestors (9)

 Incoming links (9)

 View article source

 Discussion (0)

There are no discussions about this article yet.

 Articles by others on the same topic (2)

Backpropagation by

Wikipedia Bot  1

Backpropagation is an algorithm used for training artificial neural networks. It is a supervised learning technique that helps adjust the weights of the network to minimize the difference between the predicted outputs and the actual target outputs. The term "backpropagation" is short for "backward propagation of errors," signifying its two-step process: forward pass and backward pass.

 Read the full article

Backpropagation by

Ciro Santilli  40 Updated 2025-07-16

What is backpropagation really doing? by 3Blue1Brown (2017)

Source. Good hand-wave intuition, but does not describe the exact algorithm.

 Read the full article

  See all articles in the same topic Create my own version