Feedforward neural network

A feedforward neural network composes layers without directed cycles, passing each layer's activations only to later layers.

Let $S$ be the number of active inputs. Four hidden units $h_m=H(S-m+1/2)$, $m=1,2,3,4$, detect successively larger counts. A binary threshold unit with weights $(1,-1,1,-1)$ on these features and threshold $1/2$ returns one precisely for $S=1,3$. Each hidden input weight is one. This computes parity with a single hidden layer, but representability does not guarantee easy learning by backpropagation.

A hidden unit transforms input information before the final output rather than directly representing a supplied target. Nonlinear hidden features can make a classification linearly separable in feature space even when it is not separable in the original input coordinates.

Backpropagation applies the chain rule from a neural network's output toward its input to compute the gradient of a loss with respect to every parameter.

For squared loss $E={\sum }_k(t_k-z_k{)}^2/2$, let $e_k=(t_k-z_k)g_k^{\prime }(x_k)$ and $e_j=g_j^{\prime }(x_j){\sum }_k{w}_{jk}{e}_k$. The chain rule gives $\partial E/\partial w_{jk}=-e_k{z}_j$ and $\partial E/\partial w_{ij}=-e_j{z}_i$. A gradient descent step of size $\eta$ uses $\Delta w_{jk}=\eta e_k{z}_j$ and $\Delta w_{ij}=\eta e_j{z}_i$, with every derivative evaluated at the same pre-update weights.

For hidden activations $h_j=\sigma (b_j+{\sum }_r{W}_{jr}{x}_r)$ and class probabilities $p_c={\mathrm{softmax}}_c(a+Vh)$, a one-hot label $t$ has log-likelihood $\ell ={\sum }_c{t}_c\log p_c$. Put ${\delta }_c=t_c-p_c$ and ${\Delta }_j=h_j(1-h_j){\sum }_c{V}_{cj}{\delta }_c$. ThenThese follow from the chain rule and give single-observation stochastic gradient descent updates for the negative log-likelihood.