= Sigmoid-softmax network gradients
For hidden activations $h_j=\sigma(b_j+\sum_rW_{jr}x_r)$ and class probabilities $p_c=\operatorname{softmax}_c(a+Vh)$, a one-hot label $t$ has <log-likelihood> $\ell=\sum_ct_c\log p_c$. Put $\delta_c=t_c-p_c$ and $\Delta_j=h_j(1-h_j)\sum_cV_{cj}\delta_c$. Then
$$
\partial_{a_c}\ell=\delta_c,\quad\partial_{V_{cj}}\ell=\delta_ch_j,\quad\partial_{b_j}\ell=\Delta_j,\quad\partial_{W_{jr}}\ell=\Delta_jx_r.
$$
These follow from the <chain rule> and give single-observation <stochastic gradient descent> updates for the negative <log-likelihood>.
Back to article page
