= Solution
For one observation, let $\ell_i=\sum_{c=0}^1t_{ic}\log p_{ic}$ and define $\delta_{ic}=t_{ic}-p_{ic}$. Differentiating the <softmax function> and its <log-likelihood> gives $\partial\ell_i/\partial o_{ic}=\delta_{ic}$. The <sigmoid-softmax network gradients> are
$$
\boxed{\frac{\partial\ell_i}{\partial a_c}=\delta_{ic},\qquad\frac{\partial\ell_i}{\partial V_{cj}}=\delta_{ic}h_{ij},}
$$
$$
\Delta_{ij}=h_{ij}(1-h_{ij})\sum_{c=0}^1V_{cj}\delta_{ic},\qquad
\boxed{\frac{\partial\ell_i}{\partial b_j}=\Delta_{ij},\qquad\frac{\partial\ell_i}{\partial W_{jr}}=\Delta_{ij}x_{ir}.}
$$
These equations are <backpropagation>: use the <chain rule> through the output affine map, then $\sigma'(u)=\sigma(u)(1-\sigma(u))$ through the hidden units, then the input affine map.
For a learning rate $\eta>0$, select one observation and update every parameter by $\theta\leftarrow\theta+\eta\nabla\ell_i(\theta)$, computing all entries of the <gradient> from the old parameters before applying the update. A uniformly selected observation gives an unbiased gradient of the average log-likelihood; multiplying by 3000 would instead estimate the gradient of the sum. Equivalently perform <stochastic gradient descent> on the negative log-likelihood. Batch size one makes one update per observation, and five epochs make five passes through the data, typically with a fresh random ordering on each pass.
Back to article page
