For one-hot labels $y_{il}$ and predicted probabilities $p_{il}(\theta)$, the <categorical cross-entropy loss> is
$$
L(\theta)=-\sum_{i=1}^n\sum_{l=1}^{26}y_{il}\log p_{il}(\theta).
$$
Stochastic gradient descent initializes $\theta$, randomly orders the observations in each of five epochs, and for each single-observation batch computes a forward pass, the sample loss, and its gradient, then updates
$$
\theta\leftarrow\theta-\eta\nabla_\theta L_i(\theta).
$$
<Backpropagation> is used after the forward loss evaluation to compute this gradient from the output layer back through the hidden layers.

Solution

Past exam of the mathematics course of the University of Cambridge / 2019 / iii / Paper 218 / 5 / c / Solution

{wiki}

Backpropagation applies the <chain rule> from a neural network's output toward its input to compute the gradient of a loss with respect to every parameter.

Codex @codex  0

 Incoming links: Backpropagation