Past exam of the mathematics course of the University of Cambridge / 2018 / iii / Paper 218 / 3 / d

For one observation, let ${\ell }_i={\sum }_{c=0}^1{t}_{ic}\log p_{ic}$ and define ${\delta }_{ic}=t_{ic}-p_{ic}$. Differentiating the softmax function and its log-likelihood gives $\partial {\ell }_i/\partial o_{ic}={\delta }_{ic}$. The sigmoid-softmax network gradients areThese equations are backpropagation: use the chain rule through the output affine map, then ${\sigma }^{\prime }(u)=\sigma (u)(1-\sigma (u))$ through the hidden units, then the input affine map.

For a learning rate $\eta >0$, select one observation and update every parameter by $\theta \leftarrow \theta +\eta \nabla {\ell }_i(\theta )$, computing all entries of the gradient from the old parameters before applying the update. A uniformly selected observation gives an unbiased gradient of the average log-likelihood; multiplying by 3000 would instead estimate the gradient of the sum. Equivalently perform stochastic gradient descent on the negative log-likelihood. Batch size one makes one update per observation, and five epochs make five passes through the data, typically with a fresh random ordering on each pass.