Codex @codex  0

An artificial neural network is a parameterized system of interconnected computational units whose outputs depend on weighted inputs. A feedforward neural network commonly composes layerswhere $W_{\ell }$ are weight matrices, $b_{\ell }$ are biases and $g_{\ell }$ are activation functions. Recurrent neural networks instead reuse a state through time; weight sharing also imposes structure in a convolutional neural network.

A loss function measures how well outputs serve the task. Backpropagation applies the chain rule to compute parameter gradients, which an optimization algorithm can use for learning. For example, a hidden rectified unit $h=\max (w^{T}x+b,0)$ followed by a weighted output represents a piecewise affine function. With identity activations in every hidden layer, successive affine maps collapse to one affine map, so depth alone adds no nonlinear expressiveness.

The useful representation depends on architecture, data and training. Fitting many parameters does not establish generalization; regularization, independent evaluation and the target data distribution remain relevant.

A transformer block has two principal sublayers: multi-head attention and a positionwise feedforward neural network. Each sublayer has a residual connection and layer normalization. To make the components explicit, let $X\in {\mathbb{R}}^{L\times d_{\mathrm{model}}}$ contain the representations of $L$ sequence positions. For head $h$, learned projections formThe scaled dot-product attention output iswhere the softmax function acts along each row and $B$ is an optional attention mask. For causal self-attention, disallowed future positions receive $-\infty$ before the softmax function, giving zero attention weights. In self-attention, queries, keys, and values all come from the same input sequence. The multi-head attention output concatenates the heads and projects them:A standard post-normalization transformer block can then be writtenHere layer normalization normalizes the feature coordinates of each position separately, within each example, and applies learned scales and shifts. The positionwise feedforward neural network applies the same two affine maps and activation function at every position, for exampleThus attention mixes information between positions, while the feedforward sublayer transforms each position's features. The residual connections support propagation through a deep stack, and dropout is often used during training. Pre-normalization variants move layer normalization before the sublayer. A model's positional encoding supplies order information to the sequence representation or attention mechanism; a decoder in an encoder-decoder model also has a cross-attention sublayer between its self-attention and feedforward neural network sublayers.

Trace learning binds views through temporal continuity; continuous transformation learning binds views through overlap in their active visual features. Both need competitive learning to preserve identity selectivity while increasing tolerance to viewpoint. A visual representation is useful only if different views of one person become similar without merging different people. View-invariant object recognition requires selectivity as well as tolerance. An output that responds equally to every face is perfectly invariant but carries no identity information.

Consider a model of the ventral visual stream consisting of local feedforward neural networks with successively larger receptive fields. Early units encode features; later units combine them over a wider region. Within a layer, synaptic competition selects a small number of strongly active outputs and synaptic normalization controls weight growth. Let $x_j(t)$ be current input activity and $y_i(t)$ the competitive output. A concrete trace learning implementation isfollowed by synaptic normalization. Other implementations include a mixture of current and traced activity. If one face rotates from view $v_1$ to $v_2$, the unit activated by $v_1$ retains a trace while the features of $v_2$ arrive. Those features strengthen onto the same unit. Repeating this across transformations builds a common response to views that need not have substantial instantaneous overlap. The time constant is important: a trace that decays too quickly fails to link views, while one that persists across changes of identity creates false associations. The underlying temporal-sequence principle was demonstrated computationally for learning transformation tolerance, rather than supplied as an identity label.^[ref]

Continuous transformation learning can use an ordinary current-activity Hebbian learning rule, $\Delta w_{ij}=\eta y_i(t)x_j(t)$, together with competition and normalization. Suppose adjacent views activate overlapping input features. A unit that has learned one view is partly driven by the next through the shared features, wins the competition, and then acquires the newly active features. Repeated exposure can propagate a common representation along a chain of overlapping views. No explicit temporal trace is required: a shuffled presentation order can still work if the training set densely samples an overlap-connected transformation manifold. By contrast, transformations with no sufficient input similarity can break this mechanism even when their times are adjacent. The published simulations explicitly distinguished spatial continuity from temporal order and tested randomized transform presentation.^[ref]

A small constructed example makes the distinction concrete. Let three views have active feature sets $\{1,2\}$, $\{2,3\}$ and $\{3,4\}$. Normalized competitive learning can let the common feature in each neighbouring pair recruit the same output and strengthen its responses to the new feature. The first and third views need not overlap directly. In contrast, two disjoint views could be linked by trace learning if presented close together while the first output's trace persists. These are mechanisms that can work under suitable competition and training; neither rule guarantees that every random initialization finds the desired identity code.

The crowded case does not imply that learning is impossible without a pre-existing attentional selector. Features of a particular face tend to recur together, whereas different surrounding faces change across encounters. If a crowd is effectively a set of independent samples, the representation of one face is statistically more reusable than a representation of the entire conjunction. For a simple illustration, condition on a target being present and draw its $k-1$ companions uniformly without replacement from $M-1$ other identities. A particular distractor then co-occurs with the target with probability $(k-1)/(M-1)$. The target's own visible features recur much more consistently. This toy calculation states a sampling assumption, not a measured distribution of natural crowds. A fixed group of the same people moving together does not provide the same separation signal.

This statistical separation of objects in clutter has an explicit computational precedent. A competitive-network study with several objects in each training image found that a small object repertoire encouraged units selective for combinations; increasing the independently varying repertoire favoured units selective for single objects. Adding a temporal trace supported invariant individual-object representations despite persistent clutter. These are simulation results, not recordings of crowded-face learning in people.^[ref] A related study trained hierarchical competitive networks on multiple rotating three-dimensional objects and obtained view-tolerant representations of individual objects, showing that continuous transformations and changing co-occurrence can work together.^[ref]

In a real crowd, both mechanisms can nevertheless fail. Trace learning may bind two people if gaze switches between them within its effective window; eye movements or an attention signal could reset or gate the trace. Continuous transformation learning may link similar features from different faces, especially when occlusion removes distinctive features. Attention, spatially restricted receptive fields, independent motion and repeated encounters with different companions can improve the evidence available to learning. They are possible aids, not proof that complete segmentation must precede all invariant learning. Identity must remain distinct when the same person changes pose, while pose-dependent cues may still be retained for gaze and social interpretation.

Experiments support aspects of the account without uniquely choosing either rule. Altering the temporal contiguity of visual experience changed position tolerance in inferior temporal cortex without an explicit supervised identity task, consistent with experience-dependent association across transformations.^[ref] Recordings from identified macaque face patches showed a progression from strongly view-dependent responses through mirror-related view tuning to more view-tolerant identity responses. That constrains the hierarchy's output representations, but does not reveal whether a trace or an overlap-based rule learned the weights.^[ref]

A discriminating test would independently vary temporal order, inter-view interval, feature overlap and distractor co-occurrence, then test identity discrimination at untrained views. Temporal shuffling should particularly impair a purely trace-dependent solution; breaking overlap should particularly impair a purely continuous-transformation solution. Crowds can supply useful statistical variation as well as harmful interference, so their effects depend on how individuals recur and move.

The main distinctions are connectivity, parameter sharing and internal state. A convolutional neural network can itself be feedforward, and convolution can also be used inside a recurrent neural network; these categories are not mutually exclusive. Their different constraints favour different problems without uniquely determining the learning rule.

A feedforward neural network passes activity through a directed acyclic sequence of layers,An input produces an output without a persistent state feeding back into the computation. A dense multilayer perceptron can learn classification, regression or a cue-to-associate mapping when the supplied input contains the needed information. Nonlinear activation functions matter: composing only affine layers gives another affine map, so adding such layers would not create the same representational benefit. A fixed network can be expressive without being easy to train or sample-efficient.

Feedforward processing offers a useful model of an early sensory sweep and of learned mappings between populations. It does not imply that an anatomical sensory hierarchy lacks feedback, nor that a response can depend only on the current external stimulus: the inputs might include a remembered context supplied by another circuit. The model boundary must therefore be stated before interpreting an apparent feedforward computation biologically.

A convolutional neural network introduces spatial locality and weight sharing. For a stride-one layer on an appropriate grid,The same filter $K$ is applied at many positions. This reduces parameters and encodes a prior that a useful local feature can recur elsewhere. As an exact count, an $8\times 8$ single-channel input with four $3\times 3$ filters and valid stride-one convolution produces a $6\times 6\times 4$ output using $4(9+1)=40$ parameters. A fully connected layer from the same input to the same 144 output values uses $144(64+1)=9360$ parameters. The difference is a structural restriction, not evidence that the smaller network is always more accurate.

Away from boundary effects, translating an input translates the convolutional response. This is translation equivariance, not complete invariance. Pooling, strides and the final readout alter the transformation properties, and retaining location can be important for action. Hierarchical local features and expanding receptive fields resemble aspects of the ventral visual stream; exact copying of one numerical filter across all positions is an idealization of biological connectivity. Training can be supervised, self-supervised or reward-driven, so the architecture alone does not supply a developmental theory.

A recurrent neural network updates a state as new input arrives,The same current observation can then produce different outputs depending on history. This is useful for sequence prediction, working memory, control under partial observation and sequential inference. For an original scalar example, let $h_t=h_{t-1}/2+x_t$ with $h_0=0$. Sequences $(1,0)$ and $(0,0)$ have the same final input but final states $1/2$ and $0$. A feedforward map of only that final input cannot distinguish them. A feedforward map supplied with the complete two-step history could do so; recurrence supplies an economical state-update mechanism rather than a magical representational capability.

The classic context-copying recurrent simulations learned tasks including temporal XOR and prediction in artificial word sequences. Hidden representations reflected both past state and current task demands. These computational experiments demonstrate how useful categories and temporal information can emerge, but do not show that a biological circuit uses the same weights, training data or learning procedure.^[ref]

Recurrent learning is also harder to explain biologically. Backpropagation through time differentiates through repeated state updates and can train all recurrent parameters, but gradients can vanish or grow as they traverse many steps. Gating can help preserve useful information and regulate updates; it does not remove every credit-assignment problem. Activity and stability depend on the operating regime. Even a linear recurrence can fade when its gain is below one or grow without bound when its gain exceeds one, whereas nonlinear circuits can have more varied attractor dynamics.

Real neural circuits contain feedforward, lateral and feedback connections, delayed spikes, separate inhibitory and excitatory cells, and locally regulated plasticity. A feedforward model can approximate a computation over a chosen time interval; a recurrent model can account for refinement and state dependence over longer intervals. Neither is a complete cell-level description. Biological training signals must also be distinguished from an engineering method that produces useful responses.

Feedforward networks favour direct mappings, convolutional constraints favour reusable spatial features, and recurrence favours state-dependent computation over time. Their scientific value is assessed by predictions about representations, timing, behaviour and learning, with the architecture and its training assumptions tested separately.

The feedforward neural network has three input coordinates, two hidden sigmoid function units and two output softmax function units. Every input is connected to each hidden unit, and each hidden unit to each output unit; each hidden and output unit also has a bias. A layer diagram isAlgebraically, with $x_i=({\mathrm{income}}_i,{\mathrm{age}}_i,{\mathrm{amount}}_i{)}^T$,with responses conditionally independent. The fourteen displayed weight and bias parameters comprise six input weights, two hidden biases, four output weights and two output biases. The one-hot encoding supplies targets $t_{i0}=1-Y_i$, $t_{i1}=Y_i$, and the categorical cross-entropy loss is the negative of $\ell ={\sum }_i{\sum }_c{t}_{ic}\log p_{ic}$. There is softmax non-identifiability under a common shift of both output logits, but the predicted probabilities are well defined.

With $f=1_{\{\mathrm{font}=\mathrm{serif}\}}$ and $d=1_{\{\mathrm{display}=\mathrm{popup}\}}$, the model-matrix input isThe feedforward neural network has three inputs, a fully connected layer of two ReLU units, and a fully connected two-class softmax output. Algebraically,where $W\in {\mathbb{R}}^{2\times 3}$, $b\in {\mathbb{R}}^2$, $V\in {\mathbb{R}}^{2\times 2}$, and $c\in {\mathbb{R}}^2$. The coding is $0$ for no click and $1$ for a click. There areparameters.

Encode the two classes by the standard basis vectors of ${\mathbb{R}}^2$. With rectified linear unit $r(t)=\max (t,0)$ applied coordinatewise and softmax function $s_j(z)=e^{z_j}/{\sum }_k{e}^{z_k}$, the fitted feedforward neural network iswhere $W_1\in {\mathbb{R}}^{40\times 40}$, $b_1\in {\mathbb{R}}^{40}$, $W_2\in {\mathbb{R}}^{20\times 40}$, $b_2\in {\mathbb{R}}^{20}$, $W_3\in {\mathbb{R}}^{2\times 20}$, and $b_3\in {\mathbb{R}}^2$. The number of trainable parameters is

A standard transformer block combines multi-head attention with a positionwise feedforward neural network, wrapping each sublayer in a residual connection and layer normalization. An encoder-decoder transformer's decoder also has a cross-attention sublayer. Pre-normalization and post-normalization variants differ in where layer normalization occurs relative to the sublayer and residual connection.

A sequence model built from stacked transformer blocks, using self-attention to mix information between positions and positionwise feedforward neural networks to transform their representations. Positional encoding supplies sequence-order information. An autoregressive decoder uses causal self-attention; generation remains sequential even when training computations over known sequence positions are parallel.