Feedforward neural network

A feedforward neural network composes layers without directed cycles, passing each layer's activations only to later layers.

A layered neural network with affine maps and activation functions, usually with dense connections between successive layers. A dense layer with $n$ inputs and $m$ outputs has $m(n+1)$ parameters including biases.

Let $S$ be the number of active inputs. Four hidden units $h_m=H(S-m+1/2)$, $m=1,2,3,4$, detect successively larger counts. A binary threshold unit with weights $(1,-1,1,-1)$ on these features and threshold $1/2$ returns one precisely for $S=1,3$. Each hidden input weight is one. This computes parity with a single hidden layer, but representability does not guarantee easy learning by backpropagation.

A hidden unit transforms input information before the final output rather than directly representing a supplied target. Nonlinear hidden features can make a classification linearly separable in feature space even when it is not separable in the original input coordinates.

Backpropagation applies the chain rule from a neural network's output toward its input to compute the gradient of a loss with respect to every parameter.

Apply the chain rule to a recurrent network unrolled over its input sequence, adding contributions to parameters reused at multiple time steps. Products of state Jacobians convey effects from earlier states to later losses. This can compute exact gradients for the chosen finite unrolling, while physiological availability of the same history and error signals is a separate question.

A learning scheme replacing a transposed forward-weight matrix in the teaching signal with a separate fixed random matrix. Simulations show that adaptation of the forward weights can make this feedback useful on tested tasks. It need not equal the exact loss gradient, and successful simulations alone do not establish the mechanism in the brain.^[ref]

Exact backpropagation computes hidden teaching signals using the transpose of the current forward weight matrix. Explaining how a biological feedback pathway could know or maintain those weights is the weight transport problem. Useful approximate error correction need not satisfy exact reciprocal weight equality.^[ref]

For squared loss $E={\sum }_k(t_k-z_k{)}^2/2$, let $e_k=(t_k-z_k)g_k^{\prime }(x_k)$ and $e_j=g_j^{\prime }(x_j){\sum }_k{w}_{jk}{e}_k$. The chain rule gives $\partial E/\partial w_{jk}=-e_k{z}_j$ and $\partial E/\partial w_{ij}=-e_j{z}_i$. A gradient descent step of size $\eta$ uses $\Delta w_{jk}=\eta e_k{z}_j$ and $\Delta w_{ij}=\eta e_j{z}_i$, with every derivative evaluated at the same pre-update weights.

For hidden activations $h_j=\sigma (b_j+{\sum }_r{W}_{jr}{x}_r)$ and class probabilities $p_c={\mathrm{softmax}}_c(a+Vh)$, a one-hot label $t$ has log-likelihood $\ell ={\sum }_c{t}_c\log p_c$. Put ${\delta }_c=t_c-p_c$ and ${\Delta }_j=h_j(1-h_j){\sum }_c{V}_{cj}{\delta }_c$. ThenThese follow from the chain rule and give single-observation stochastic gradient descent updates for the negative log-likelihood.