Past exam of the mathematics course of the University of Cambridge / 2019 / iii / Paper 218 / 5 / c / Solution

For one-hot labels $y_{il}$ and predicted probabilities $p_{il}(\theta )$, the categorical cross-entropy loss isStochastic gradient descent initializes $\theta$, randomly orders the observations in each of five epochs, and for each single-observation batch computes a forward pass, the sample loss, and its gradient, then updatesBackpropagation is used after the forward loss evaluation to compute this gradient from the output layer back through the hidden layers.