Neural network

A neural network composes affine maps with nonlinear activation functions and learns their weights from data.

A binary threshold unit outputs one when its weighted input reaches its threshold and zero otherwise. Its hard threshold is not differentiable, so ordinary backpropagation cannot directly train the exact threshold model. Smooth activation functions can approximate its computation.

An activation function converts a unit's weighted input to its output. A sigmoid function, hyperbolic tangent or rectifier can give nonlinear features. Replacing all hidden activation functions by the identity collapses consecutive affine layers into a single affine map, removing the representational benefit of those hidden layers.

A recurrent network with symmetric interactions and an energy function. In the binary version, units take values $\pm 1$, self-couplings are zero, and asynchronous threshold updates can retrieve stored patterns from incomplete or corrupted cues. Fixed points are local energy minima, which need not be the intended memories or global minima.

For symmetric zero-diagonal weights and zero thresholds, update one binary unit to the sign of its current field, retaining it at a zero field. The displayed energy change is negative on every actual flip. There are finitely many states, so a fair asynchronous update schedule eventually reaches a fixed point that is a minimum against single-unit flips. Synchronous updates can instead cycle, and local minima need not be global minima.

For $M$ binary units and desired patterns ${\xi }^{\mu }$, use these symmetric weights and set $w_{ii}=0$. For one pattern, the field at that pattern is $(M-1){\xi }_i/M$, so it is stable. Several patterns add cross-talk terms; storage by this rule alone does not guarantee stability of every arbitrary collection of patterns.

A feedforward neural network composes layers without directed cycles, passing each layer's activations only to later layers.

Let $S$ be the number of active inputs. Four hidden units $h_m=H(S-m+1/2)$, $m=1,2,3,4$, detect successively larger counts. A binary threshold unit with weights $(1,-1,1,-1)$ on these features and threshold $1/2$ returns one precisely for $S=1,3$. Each hidden input weight is one. This computes parity with a single hidden layer, but representability does not guarantee easy learning by backpropagation.

A hidden unit transforms input information before the final output rather than directly representing a supplied target. Nonlinear hidden features can make a classification linearly separable in feature space even when it is not separable in the original input coordinates.

Backpropagation applies the chain rule from a neural network's output toward its input to compute the gradient of a loss with respect to every parameter.

For squared loss $E={\sum }_k(t_k-z_k{)}^2/2$, let $e_k=(t_k-z_k)g_k^{\prime }(x_k)$ and $e_j=g_j^{\prime }(x_j){\sum }_k{w}_{jk}{e}_k$. The chain rule gives $\partial E/\partial w_{jk}=-e_k{z}_j$ and $\partial E/\partial w_{ij}=-e_j{z}_i$. A gradient descent step of size $\eta$ uses $\Delta w_{jk}=\eta e_k{z}_j$ and $\Delta w_{ij}=\eta e_j{z}_i$, with every derivative evaluated at the same pre-update weights.

For hidden activations $h_j=\sigma (b_j+{\sum }_r{W}_{jr}{x}_r)$ and class probabilities $p_c={\mathrm{softmax}}_c(a+Vh)$, a one-hot label $t$ has log-likelihood $\ell ={\sum }_c{t}_c\log p_c$. Put ${\delta }_c=t_c-p_c$ and ${\Delta }_j=h_j(1-h_j){\sum }_c{V}_{cj}{\delta }_c$. ThenThese follow from the chain rule and give single-observation stochastic gradient descent updates for the negative log-likelihood.

The sigmoid functionis a smooth activation function with derivative ${\sigma }^{\prime }(x)=\sigma (x)(1-\sigma (x))$.

The standard logistic function maps a log odds $z$ to probability $1/(1+e^{-z})$. More generally, its translated and scaled form is $L/(1+e^{-k(x-x_0)})$.

The rectified linear unit activation is $\mathrm{ReLU}(x)=\max (x,0)$.

The Gaussian error linear unit activation is $\mathrm{GELU}(x)=x\Phi (x)$, where $\Phi$ is the standard normal distribution function.

For logits $z_1,\dots ,z_K$, the softmax probabilities are $p_k=e^{z_k}/{\sum }_j{e}^{z_j}$.

Adding the same scalar-valued function of the input to every class logit leaves all softmax probabilities unchanged. A reference-class constraint removes this common-shift non-identifiability.

For one-hot label $y$ and predicted class probabilities $p$, categorical cross-entropy is $-{\sum }_k{y}_k\log p_k$, the negative categorical log-likelihood.