Codex @codex  0

Inception is a family of convolutional neural network architectures built around modules with parallel processing branches. An early module combines several convolution sizes and a pooling layer, then joins their outputs along the channel dimension. Pointwise convolution can reduce channel counts before more expensive spatial operations.^[ref]

For a stipulated shape example, let every branch produce a $28\times 28$ grid, with output channel counts $64$, $128$, $32$ and $32$. Joining channels gives a $28\times 28\times 256$ output becauseThe equal spatial grid sizes are an assumption ensured by the branch design. Channel joining does not add corresponding values into one channel; it retains the distinct branch outputs as separate channels of the combined neural network feature map.

A cost example explains the reduction branch. Ignoring biases, a dense three-by-three convolutional layer from $192$ input channels to $128$ output channels hasweights. Reducing first from $192$ to $32$ channels with a one-by-one pointwise convolution, then applying the three-by-three layer, usesweights. This own example isolates channel dimensions; it is not a reported layer configuration or a claim that the two computations represent identical functions.

The original paper presents this design in the GoogLeNet network and evaluates image classification and detection. Later Rethinking the Inception Architecture for Computer Vision by Christian Szegedy and others develops additional factorization and training choices. The family name therefore does not identify one immutable list of layers or one universal performance result.^[ref][ref]

A neural network channel indexes a component or response type at each position of a neural network's data grid. In a two-dimensional convolutional neural network, the channel axis is distinct from the two spatial axes. For example, an image represented with the RGB color model has red, green and blue input channels, while an intermediate neural network feature map can have many learned response channels.^[ref]

The arrangement of axes must be stated. In the documented convention $(N,C,H,W)$, $N$ counts examples, $C$ channels, $H$ rows and $W$ columns. Thus two three-channel images, each four pixels high and five pixels wide, have shape $(2,3,4,5)$ and containscalar entries. Changing the number of examples does not change the number of channels in each example.^[ref]

A dense convolutional layer can mix all input channels into each output channel. For the preceding input, use eight output channels, three-by-three filters, convolution stride one and no padding. The output shape iswith $96$ scalar entries. Its eight filters use $8(3)(3)(3)=216$ weights, plus eight biases if enabled, giving $224$ parameters. The leading example count does not multiply this parameter count: the same weights process each example.^[ref]

The eight output channels are not eight new primary colors. Their values are determined by learned combinations of the preceding channels and spatial neighborhoods. Individual channels can be useful for interpretation, but a channel's index does not by itself identify a human-named feature or make its response a probability.

A pointwise convolution makes the channel operation especially clear: at every spatial point it maps one channel vector to another using the same matrix of weights. By changing that matrix, it can reduce or increase the channel count while preserving the grid dimensions under unit-stride, unpadded conditions. Keeping channel, spatial and example axes separate prevents both shape errors and misleading interpretations.

A neural network receptive field describes which input positions can influence a specified output unit through the network's computational connections. A local convolutional layer uses only a small input region for each output, but successive layers can combine information from a larger region. This dependence is relative to a chosen input layer; an immediate previous-layer neighborhood and the corresponding original-input region have different sizes.^[ref]

For an own one-dimensional example, use two kernel-size-three convolutional layers, with convolution stride one, no padding and no convolution dilation. Intermediate units $0$, $1$ and $2$ depend on input sets $\{0,1,2\}$, $\{1,2,3\}$ and $\{2,3,4\}$ respectively. The first output of the second layer combines these three units, so its possible original-input dependency isTwo size-three filters therefore give a size-five receptive region, not size nine.

A recurrence tracks the geometry of an undilated serial network. Let $r_{\ell }$ be the span of the receptive region in original-input positions and $j_{\ell }$ the original-input index separation between adjacent layer-$\ell$ outputs. With kernel size $k_{\ell }$, stride $s_{\ell }$ and initial values $r_0=j_0=1$,The last contributing previous-layer unit lies $(k_{\ell }-1)j_{\ell -1}$ input positions beyond the first, which proves the span increment. This derivation assumes the stated local serial connectivity, no padding and no dilation.

If the first size-three layer instead has stride two, then $r_1=3$ and $j_1=2$. A second size-three, stride-one layer has $r_2=7$: its three input supports are $\{0,1,2\}$, $\{2,3,4\}$ and $\{4,5,6\}$. Their union reaches from zero through six. For other sampling choices, a span can include gaps, so span and number of contributing positions need not agree.

This is a structural dependency calculation. Zero weights or the current behavior of an activation function can suppress an influence that the architecture permits; the calculation does not measure how strongly each position affects a particular output. A unit-stride pointwise convolution mixes neural network channels at the same location and does not enlarge this spatial region. The distinction helps explain why many channels alone do not guarantee access to distant input context.

A pointwise convolution uses a spatial kernel of size one to combine neural network channels independently at each spatial position. For a two-dimensional input this is a one-by-one convolutional layer. In the ordinary dense-channel, unit-stride case, the same affine map acts on the channel vector at every grid point.^[ref]

Let $x_{i,j}\in {\mathbb{R}}^{C_{\mathrm{in}}}$ be the channel vector at position $(i,j)$. With weight matrix $M$ and neural network bias vector $b$, the output isThis is matrix multiplication across channels, not an average of neighboring spatial positions. The same $M$ and $b$ are reused throughout the grid.

For an example without biases, takeAt a point with input $(1,2,3{)}^{\mathsf{T}}$ the output is $(5,1{)}^{\mathsf{T}}$. At another point with input $(3,0,4{)}^{\mathsf{T}}$ it is $(3,4{)}^{\mathsf{T}}$. A three-channel neural network feature map has become a two-channel map without mixing the two positions.

There are $C_{\mathrm{in}}{C}_{\mathrm{out}}$ weights, plus $C_{\mathrm{out}}$ optional biases. Thus three input and two output channels require six weights, or eight parameters including biases. A dense three-by-three filter with the same channel counts requires $9(3)(2)=54$ weights, or fifty-six parameters with biases. This comparison counts parameters for the stated connectivity; it does not claim that the two layers compute equivalent functions.

Channel reduction can make a later spatial convolutional layer cheaper. The original Inception uses one-by-one reductions before some larger filters. A pointwise operation can also increase the channel count; size one refers to its spatial footprint, not to the number of channels.^[ref] With no intervening spatial operation, pointwise layers and pointwise activation functions alone do not enlarge the spatial neural network receptive field.