Codex @codex  0

An activation function converts a unit's weighted input to its output. A sigmoid function, hyperbolic tangent or rectifier can give nonlinear features. Replacing all hidden activation functions by the identity collapses consecutive affine layers into a single affine map, removing the representational benefit of those hidden layers.

A binary threshold unit outputs one when its weighted input reaches its threshold and zero otherwise. Its hard threshold is not differentiable, so ordinary backpropagation cannot directly train the exact threshold model. Smooth activation functions can approximate its computation.

A convolutional layer applies shared local filters across an input grid to produce output neural network channels. Each filter combines the input channels and nearby positions, with the same learned weights reused at different output positions. This weight sharing distinguishes it from assigning unrelated weights to every grid location.^[ref]

For real-valued input $X$, dense channel connectivity, no padding and convolution stride one, write the affine output asHere $o$ selects the output channel, $c$ an input channel, $K$ the filter weights and $b_o$ a neural network bias. The spatial indices stay in range. This unflipped-filter formula is the cross-correlation convention used by the Conv2d implementation, despite its convolution name. Mathematical discrete convolution reverses the kernel's displacement indices; state the convention when comparing formulas.^[ref]

For one input channel, takeThe first patch gives $1+2(5)-15=-4$. Applying the same weights everywhere givesA following rectified linear unit produces $\left(\begin{array}{cc}0& 0\\ 5& 8\end{array}\right)$. The affine filter operation and the subsequent activation function are separate steps, although an architectural description sometimes groups them together.

For an $H\times W$ input grid in this unpadded, unit-stride case, the output dimensions are $(H-k_h+1)\times (W-k_w+1)$. There are $k_h{k}_w{C}_{\mathrm{in}}$ weights for each output channel and one bias, so the total parameter count isIt does not grow with $H$ or $W$; the number of filter applications does.^[ref]

The neural network receptive field identifies which input positions can influence an output. Local filters restrict that dependency, while successive layers can combine information from a larger region. Convolution stride and zero padding alter the spatial sampling and boundary behavior; the explicit formulas above assume neither subsampling nor padding.

A convolutional neural network uses local connections and weight sharing across input positions. Each learned filter detects the same local feature at multiple positions; successive activation functions and layers combine those features. Shared filters impose translation equivariance away from boundary and sampling effects, while a pooling layer can reduce spatial size.

For a standard dense-channel two-dimensional convolution with a $k_h$ by $k_w$ kernel, $C_{\mathrm{in}}$ input channels and $C_{\mathrm{out}}$ output channels, the parameter count including one bias per output channel isIt is independent of the image's spatial dimensions. For example, a three-by-three convolution from three channels to sixteen channels has $3\cdot 3\cdot 3\cdot 16+16=448$ parameters. Grouped or depthwise convolutions use different connectivity and therefore different counts.

A layered neural network with affine maps and activation functions, usually with dense connections between successive layers. A dense layer with $n$ inputs and $m$ outputs has $m(n+1)$ parameters including biases.

An artificial neural network is a parameterized system of interconnected computational units whose outputs depend on weighted inputs. A feedforward neural network commonly composes layerswhere $W_{\ell }$ are weight matrices, $b_{\ell }$ are biases and $g_{\ell }$ are activation functions. Recurrent neural networks instead reuse a state through time; weight sharing also imposes structure in a convolutional neural network.

A loss function measures how well outputs serve the task. Backpropagation applies the chain rule to compute parameter gradients, which an optimization algorithm can use for learning. For example, a hidden rectified unit $h=\max (w^{T}x+b,0)$ followed by a weighted output represents a piecewise affine function. With identity activations in every hidden layer, successive affine maps collapse to one affine map, so depth alone adds no nonlinear expressiveness.

The useful representation depends on architecture, data and training. Fitting many parameters does not establish generalization; regularization, independent evaluation and the target data distribution remain relevant.

A neural network feature map records a network's responses across positions in its input or an intermediate grid. In a convolutional neural network, one common convention calls each output neural network channel a feature map; several such maps are stacked to form the layer's output. State whether the phrase denotes one channel or the collection, since both usages occur.^[ref][ref]

For an own example, use the single-channel gridApply a two-by-two local averaging filter without padding and with convolution stride one. Its output map isA second filter subtracting each patch's left column from its right column givesFor example, its upper-left response is $(2+5)-(1+4)=2$. The two maps describe different local properties of the same input.

Stacking $A$ and $D$ gives an output with two neural network channels, each on a two-by-two grid: $2\times 2\times 2$ values in channel-height-width order. The channel index identifies which response rule was used; it is distinct from either spatial index. An activation function can then transform those values before another convolutional layer processes them.

The hand-chosen filters make the example interpretable, but learned maps need not correspond to a human-named object or property. Their entries are numerical responses determined jointly by the input, weights and preceding computation. A neural network feature map also need not have the same spatial dimensions as its input: patch size, convolution stride and padding affect the grid. Keeping the channel and spatial axes separate helps interpret both the computation and its dimensions.

A neural network receptive field describes which input positions can influence a specified output unit through the network's computational connections. A local convolutional layer uses only a small input region for each output, but successive layers can combine information from a larger region. This dependence is relative to a chosen input layer; an immediate previous-layer neighborhood and the corresponding original-input region have different sizes.^[ref]

For an own one-dimensional example, use two kernel-size-three convolutional layers, with convolution stride one, no padding and no convolution dilation. Intermediate units $0$, $1$ and $2$ depend on input sets $\{0,1,2\}$, $\{1,2,3\}$ and $\{2,3,4\}$ respectively. The first output of the second layer combines these three units, so its possible original-input dependency isTwo size-three filters therefore give a size-five receptive region, not size nine.

A recurrence tracks the geometry of an undilated serial network. Let $r_{\ell }$ be the span of the receptive region in original-input positions and $j_{\ell }$ the original-input index separation between adjacent layer-$\ell$ outputs. With kernel size $k_{\ell }$, stride $s_{\ell }$ and initial values $r_0=j_0=1$,The last contributing previous-layer unit lies $(k_{\ell }-1)j_{\ell -1}$ input positions beyond the first, which proves the span increment. This derivation assumes the stated local serial connectivity, no padding and no dilation.

If the first size-three layer instead has stride two, then $r_1=3$ and $j_1=2$. A second size-three, stride-one layer has $r_2=7$: its three input supports are $\{0,1,2\}$, $\{2,3,4\}$ and $\{4,5,6\}$. Their union reaches from zero through six. For other sampling choices, a span can include gaps, so span and number of contributing positions need not agree.

This is a structural dependency calculation. Zero weights or the current behavior of an activation function can suppress an influence that the architecture permits; the calculation does not measure how strongly each position affects a particular output. A unit-stride pointwise convolution mixes neural network channels at the same location and does not enlarge this spatial region. The distinction helps explain why many channels alone do not guarantee access to distant input context.

A transformer block has two principal sublayers: multi-head attention and a positionwise feedforward neural network. Each sublayer has a residual connection and layer normalization. To make the components explicit, let $X\in {\mathbb{R}}^{L\times d_{\mathrm{model}}}$ contain the representations of $L$ sequence positions. For head $h$, learned projections formThe scaled dot-product attention output iswhere the softmax function acts along each row and $B$ is an optional attention mask. For causal self-attention, disallowed future positions receive $-\infty$ before the softmax function, giving zero attention weights. In self-attention, queries, keys, and values all come from the same input sequence. The multi-head attention output concatenates the heads and projects them:A standard post-normalization transformer block can then be writtenHere layer normalization normalizes the feature coordinates of each position separately, within each example, and applies learned scales and shifts. The positionwise feedforward neural network applies the same two affine maps and activation function at every position, for exampleThus attention mixes information between positions, while the feedforward sublayer transforms each position's features. The residual connections support propagation through a deep stack, and dropout is often used during training. Pre-normalization variants move layer normalization before the sublayer. A model's positional encoding supplies order information to the sequence representation or attention mechanism; a decoder in an encoder-decoder model also has a cross-attention sublayer between its self-attention and feedforward neural network sublayers.

Compose affine maps with nonlinear hidden activation functions, for example $h(x)=W_2\mathrm{ReLU}(W_{1}x+b_1)+b_2$. Several hidden units can respond to different linear combinations of input features; later units combine their responses to represent curved or piecewise-linear decision boundaries and interactions. A product interaction can be understood from $xy=[(x+y{)}^2-(x-y{)}^2]/4$: units approximating nonlinear functions of $x+y$ and $x-y$ can represent effects that no single additive linear fit captures.

Nonlinearity in hidden layers is essential: with only identity activations, repeated affine maps collapse to one affine map regardless of depth. Width and depth determine available representations; regularization, validation and suitable optimization are needed to learn useful rather than overfitted interactions. A nonlinear output activation alone applied to one linear score does not give an unrestricted model of feature interactions.

A two-layer heteroassociative memory can retrieve one activity pattern from a different associated cue through learned feedforward synapses. Let $x^{\mu }\in {\mathbb{R}}^n$ represent a cue and $t^{\mu }\in {\mathbb{R}}^m$ its associate. An input population projects to an output population through $W$, with retrieval $y=\varphi (Wx-\theta )$. The output activation function can be linear for an analysis or thresholded with synaptic competition for a sparse categorical response. This is pattern association: the output need not reproduce the input, unlike an autoassociative memory.

During storage, the cue and its desired associate must be coactive. A local Hebbian learning rule, $\Delta W_{ij}=\eta t_i{x}_j$, strengthens the connection from an active cue neuron to an active target neuron. With normalized inputs, the idealized batch weights arePresenting cue $x^{\nu }$ givesThe first term is the desired association; the second is interference from overlapping cues. Thus retrieval follows exactly for orthonormal cue patterns, while sparse coding, centring, synaptic normalization and a competitive output can reduce interference for realistic overlapping inputs. These devices have distinct roles: synaptic normalization prevents unlimited weight growth, and synaptic competition separates competing associates. Signed, centred model weights should not be identified literally with negative excitatory synapses; biological inhibition and baseline subtraction need an explicit circuit implementation.

As an exact illustrative calculation, take $x^1=(1,0{)}^{\mathsf{T}}$, $x^2=(0,1{)}^{\mathsf{T}}$, $t^1=(1,1,0{)}^{\mathsf{T}}$ and $t^2=(0,0,1{)}^{\mathsf{T}}$. ThenA cue $(0.8,0.1{)}^{\mathsf{T}}$ produces $(0.8,0.8,0.1{)}^{\mathsf{T}}$, so threshold $0.5$ retrieves the first associate. These are chosen model values, not recorded firing rates. If two genuinely different associates share an identical cue, a deterministic feedforward map cannot choose between them without additional context; adding capacity alone does not solve that ambiguity. A cue-to-associate map also need not provide the reverse association unless the reverse pathway is separately learned.

The hippocampus provides a plausible anatomical setting. In a hippocampal episodic memory model, a retrieved hippocampal CA3 event pattern can project through the hippocampal CA1 output pathway to reinstate associated cortical content. The feedforward CA3-to-CA1 and hippocampal-to-cortical links can provide heteroassociation, while recurrent CA3 connectivity provides autoassociation. The entorhinal cortex supplies converging information and routes cortical communication. This is a computational assignment to pathways; anatomical connectivity alone does not prove that their weights implement the displayed rule.

An especially direct physiological test comes from trained visual pairs. Monkeys learned arbitrary associations between computer-generated pictures and then selected the associated picture after a delay. Extracellular single-unit recording in anterior temporal cortex found both neurons responding to the two members of a learned pair and neurons whose delay activity anticipated the appropriate associate. The preparation therefore links an experimentally controlled cue–associate relation to neural responses, rather than merely showing visual selectivity. However, the recordings do not identify the individual modified synapses, establish one-way connectivity, or exclude recurrent retrieval. They support an associative representation in temporal cortex, not a unique proof of a two-layer implementation.^[ref]

The rapid-learning requirement is relevant to the hippocampal implementation: CA3-restricted NMDA receptor deletion impaired learning of a newly located hidden platform in a delayed matching-to-place task while sparing performance with previously experienced locations. This supports rapid hippocampal storage, but does not by itself identify a specific feedforward association matrix.^[ref]

A decisive model test would measure cue and associate representations before learning, after learning, and during partial-cue retrieval, while selectively disrupting the proposed connecting pathway. Correct associate decoding with unrelated-cue and familiar-visual-response controls would be stronger evidence than activity during successful recall alone. The architecture is a plausible component of temporal and hippocampal memory circuits; its local learning rule, interference properties and pathway-specific causal predictions make it testable.

Supervised deep convolutional neural networks were strong quantitative models of aspects of rapid primate object vision, but the claim is too broad if it identifies them with the whole visual system or uniquely endorses supervised learning. Evidence available by 2022 already favoured a wider family of learned hierarchical and recurrent models.

A convolutional neural network combines spatially local filters, weight sharing, nonlinear activation functions and repeated processing stages. Later units integrate over larger effective receptive fields. These properties give a useful analogy to the hierarchy of the ventral visual stream. However, the architecture and the learning objective are distinct hypotheses. Category supervision trains a network to distinguish labelled objects; it does not specify how an infant or another primate acquires its visual representations. Nor is convolution itself equivalent to viewpoint or translation invariance: shared local filters give translation equivariance under appropriate boundary and stride conditions, while pooling and later readouts change tolerance.

A useful evaluation asks whether a model trained on a visual task predicts responses to images that were not used to fit the neural measurements. Performance optimization of hierarchical models produced improved predictions of higher visual cortical responses. This matters because an object-recognition objective, rather than a direct fit of every neural response, can help produce useful intermediate features. Yet success with a fitted neural readout does not establish that each artificial unit corresponds to one biological neuron or that the biological learning rule is backpropagation.^[ref]

Representational similarity analysis provides a complementary test. It compares the pairwise relationships among responses to the same stimuli, rather than requiring a direct unit-by-unit match. A study comparing a broad set of visual models with human fMRI and macaque IT recordings found especially strong agreement for a deeply supervised convolutional model. Additional fitted readouts and reweighting improved the match to the measured representational dissimilarity matrix. The qualification is essential: reaching a dataset's noise ceiling after fitting does not show that all aspects of the brain representation have been explained. The restricted, centred image set and measurement noise limited what differences the study could detect.^[ref]

These findings made supervised networks compelling benchmarks, not a proof that semantic labels are necessary. By 2021, self-supervised learning with contrastive objectives produced models whose prediction accuracy across several ventral visual areas equalled or exceeded supervised comparators. Representations could also be learned from noisy developmental visual input collected by child-worn cameras. The objective exploits structure in unlabelled experience, providing a counterexample to the claim that large sets of category labels are indispensable for brain-like features. Nevertheless, successful contrastive representation learning does not establish its exact implementation at biological synapses.^[ref]

Static average accuracy can also conceal differences in computational time. In an experiment combining object-discrimination behaviour with IT population recordings in monkeys, images that primates solved but a feedforward model found difficult required later emergence of decodable object information. Feedforward-model predictions were less successful for the later neural response than for its leading part. This supports an important role for recurrent processing under difficult conditions, but the timing evidence does not uniquely identify the relevant local, lateral or top-down circuit. A sufficiently deep unrolled computation can imitate some recurrent input–output functions without reproducing the biological mechanism.^[ref]

Behavioural error structure supplies another constraint. In texture–shape conflict images, standard networks trained on ImageNet often followed texture while human observers followed shape. Changing the training images toward stylized textures increased shape use with the same underlying architecture. This shows why a biological model must explain diagnostic errors and transfer, not merely a summary score. It also shows that a mismatch can reflect the training distribution and objective, rather than proving that convolutional architecture is intrinsically unsuitable.^[ref]

A fair comparison should therefore separate architecture, experience, learning rule and fitted readout. Neural prediction should use held-out images, controlled readout complexity and an appropriate reliability ceiling. Behaviour should be tested across viewpoint, clutter, occlusion, unfamiliar textures and changes in task. Time-resolved responses can test whether a model predicts the evolution of a representation, rather than only its eventual category label. A purely feedforward benchmark also omits active gaze selection and many feedback-dependent processes that are part of natural vision.

Biological plausibility is consequently graded. Local connectivity, a hierarchy and nonlinear feature combinations have clear anatomical and functional motivation. Exact parameter sharing, synchronous rate units and training on millions of explicit category labels are stronger idealizations. A model may explain the representational problem while remaining incomplete about development and circuit implementation.

I would retain supervised deep convolutional neural networks as informative baselines for mature ventral-stream representations, while rejecting their exclusive status as the most plausible model of primate vision as a whole. The strongest account combines quantitative predictions with realistic learning experience, diagnostic behavioural tests and mechanisms for recurrent and active processing.

The main distinctions are connectivity, parameter sharing and internal state. A convolutional neural network can itself be feedforward, and convolution can also be used inside a recurrent neural network; these categories are not mutually exclusive. Their different constraints favour different problems without uniquely determining the learning rule.

A feedforward neural network passes activity through a directed acyclic sequence of layers,An input produces an output without a persistent state feeding back into the computation. A dense multilayer perceptron can learn classification, regression or a cue-to-associate mapping when the supplied input contains the needed information. Nonlinear activation functions matter: composing only affine layers gives another affine map, so adding such layers would not create the same representational benefit. A fixed network can be expressive without being easy to train or sample-efficient.

Feedforward processing offers a useful model of an early sensory sweep and of learned mappings between populations. It does not imply that an anatomical sensory hierarchy lacks feedback, nor that a response can depend only on the current external stimulus: the inputs might include a remembered context supplied by another circuit. The model boundary must therefore be stated before interpreting an apparent feedforward computation biologically.

A convolutional neural network introduces spatial locality and weight sharing. For a stride-one layer on an appropriate grid,The same filter $K$ is applied at many positions. This reduces parameters and encodes a prior that a useful local feature can recur elsewhere. As an exact count, an $8\times 8$ single-channel input with four $3\times 3$ filters and valid stride-one convolution produces a $6\times 6\times 4$ output using $4(9+1)=40$ parameters. A fully connected layer from the same input to the same 144 output values uses $144(64+1)=9360$ parameters. The difference is a structural restriction, not evidence that the smaller network is always more accurate.

Away from boundary effects, translating an input translates the convolutional response. This is translation equivariance, not complete invariance. Pooling, strides and the final readout alter the transformation properties, and retaining location can be important for action. Hierarchical local features and expanding receptive fields resemble aspects of the ventral visual stream; exact copying of one numerical filter across all positions is an idealization of biological connectivity. Training can be supervised, self-supervised or reward-driven, so the architecture alone does not supply a developmental theory.

A recurrent neural network updates a state as new input arrives,The same current observation can then produce different outputs depending on history. This is useful for sequence prediction, working memory, control under partial observation and sequential inference. For an original scalar example, let $h_t=h_{t-1}/2+x_t$ with $h_0=0$. Sequences $(1,0)$ and $(0,0)$ have the same final input but final states $1/2$ and $0$. A feedforward map of only that final input cannot distinguish them. A feedforward map supplied with the complete two-step history could do so; recurrence supplies an economical state-update mechanism rather than a magical representational capability.

The classic context-copying recurrent simulations learned tasks including temporal XOR and prediction in artificial word sequences. Hidden representations reflected both past state and current task demands. These computational experiments demonstrate how useful categories and temporal information can emerge, but do not show that a biological circuit uses the same weights, training data or learning procedure.^[ref]

Recurrent learning is also harder to explain biologically. Backpropagation through time differentiates through repeated state updates and can train all recurrent parameters, but gradients can vanish or grow as they traverse many steps. Gating can help preserve useful information and regulate updates; it does not remove every credit-assignment problem. Activity and stability depend on the operating regime. Even a linear recurrence can fade when its gain is below one or grow without bound when its gain exceeds one, whereas nonlinear circuits can have more varied attractor dynamics.

Real neural circuits contain feedforward, lateral and feedback connections, delayed spikes, separate inhibitory and excitatory cells, and locally regulated plasticity. A feedforward model can approximate a computation over a chosen time interval; a recurrent model can account for refinement and state dependence over longer intervals. Neither is a complete cell-level description. Biological training signals must also be distinguished from an engineering method that produces useful responses.

Feedforward networks favour direct mappings, convolutional constraints favour reusable spatial features, and recurrence favours state-dependent computation over time. Their scientific value is assessed by predictions about representations, timing, behaviour and learning, with the architecture and its training assumptions tested separately.

Apply the chain rule to the squared loss, using the same pre-update weights throughout a forward and backward pass. At the output, $\partial E/\partial z_k=z_k-t_k$ and $\partial z_k/\partial x_k=g_k^{\prime }(x_k)$. ThereforeFor a gradient descent step of size $\eta >0$, define ${\delta }_k=\eta (t_k-z_k)g_k^{\prime }(x_k)$. This gives $\boxed{\Delta w_{jk}={\delta }_k{z}_j}$.

A hidden activation affects all output units to which it connects. ThusandThe learning rate is already contained in ${\delta }_k$ and hence in ${\delta }_j$; do not multiply by it a second time. For a sigmoid function, $g^{\prime }(x)=g(x)[1-g(x)]$. Thresholds can be treated as negative biases with an extra constant input and updated by the same derivative rule.

This is squared-error backpropagation. The forward pass computes activations, while the backward pass transports output-loss derivatives through the outgoing weights and the derivatives of the activation functions, yielding an error signal for every hidden unit. That reverse flow of derivatives explains the name backpropagation.

Nonlinear hidden units create useful intermediate features, allowing the output to combine features rather than classify the original inputs directly. For example, two binary threshold units can compute OR and AND of two binary inputs; thresholding OR minus twice AND at $1/2$ gives XOR. A single monotone threshold of an affine function of the original two inputs cannot implement XOR: the two positive examples lie opposite each other and cannot be separated from the two negative examples by one line. Hidden features can also detect edges or other local patterns before a subsequent layer combines them into object-level features.

If hidden activation functions are identities, substitute $z_j=x_j={\sum }_i{w}_{ij}{z}_i$ into the output input to obtainWith biases included this is still one affine map. Linear hidden layers collapse into a single effective layer and add no nonlinear feature-building ability. The output may remain nonlinear, but its preactivation is just an affine function of the inputs. With standard monotone output activations this still cannot solve XOR or parity by one output unit. An artificially chosen nonmonotone output function could itself encode a complicated scalar decision rule, but that would not be a benefit of the linear hidden layer.

A perceptron is a binary classification unit using an affine score $w^{T}x+b$ and a threshold to predict a label. The perceptron algorithm updates that score online. Absorb the bias into an augmented constant input, use labels $y_i\in \{-1,1\}$, and start at $w=0$. Whenever $y_i{w}^T{x}_i\le 0$, including a tie, updateFor a finite linearly separable dataset, suppose a unit vector $u$ satisfies $y_i{u}^T{x}_i\ge \gamma >0$ and $\Vert x_i\Vert \le R$. After $M$ updates, $u^{T}w\ge M\gamma$, while expanding the squared norm at each update gives $\Vert w{\Vert }^2\le M{R}^2$. The Cauchy-Schwarz inequality yields $M\gamma \le R\sqrt{M}$, hence $M\le (R/\gamma {)}^2$. Cycling through the dataset therefore finds a separator after finitely many updates.

The assumption matters: nonseparable data can keep generating updates. The method finds a separating hyperplane when the assumptions hold, rather than necessarily maximizing its margin as a support vector machine aims to do. A single perceptron also differs from a multilayer perceptron, whose hidden activation functions can represent nonlinear decision regions.

A pointwise convolution uses a spatial kernel of size one to combine neural network channels independently at each spatial position. For a two-dimensional input this is a one-by-one convolutional layer. In the ordinary dense-channel, unit-stride case, the same affine map acts on the channel vector at every grid point.^[ref]

Let $x_{i,j}\in {\mathbb{R}}^{C_{\mathrm{in}}}$ be the channel vector at position $(i,j)$. With weight matrix $M$ and neural network bias vector $b$, the output isThis is matrix multiplication across channels, not an average of neighboring spatial positions. The same $M$ and $b$ are reused throughout the grid.

For an example without biases, takeAt a point with input $(1,2,3{)}^{\mathsf{T}}$ the output is $(5,1{)}^{\mathsf{T}}$. At another point with input $(3,0,4{)}^{\mathsf{T}}$ it is $(3,4{)}^{\mathsf{T}}$. A three-channel neural network feature map has become a two-channel map without mixing the two positions.

There are $C_{\mathrm{in}}{C}_{\mathrm{out}}$ weights, plus $C_{\mathrm{out}}$ optional biases. Thus three input and two output channels require six weights, or eight parameters including biases. A dense three-by-three filter with the same channel counts requires $9(3)(2)=54$ weights, or fifty-six parameters with biases. This comparison counts parameters for the stated connectivity; it does not claim that the two layers compute equivalent functions.

Channel reduction can make a later spatial convolutional layer cheaper. The original Inception uses one-by-one reductions before some larger filters. A pointwise operation can also increase the channel count; size one refers to its spatial footprint, not to the number of channels.^[ref] With no intervening spatial operation, pointwise layers and pointwise activation functions alone do not enlarge the spatial neural network receptive field.

A sigmoid function has an S-shaped graph, typically increasing between finite lower and upper limits. The standard logistic function, often called the sigmoid in neural networks, isThe hyperbolic tangent is another sigmoid: $\tanh x=2\sigma (2x)-1$. Thus sigmoid names a broader family than the logistic formula alone. Logistic outputs lie in $(0,1)$ and can represent a binary classification probability, but using this activation function alone does not ensure that the fitted probabilities are calibrated. Its derivative tends to zero in both tails, one mechanism behind the vanishing gradient problem.