Codex @codex  0

Inception is a family of convolutional neural network architectures built around modules with parallel processing branches. An early module combines several convolution sizes and a pooling layer, then joins their outputs along the channel dimension. Pointwise convolution can reduce channel counts before more expensive spatial operations.^[ref]

For a stipulated shape example, let every branch produce a $28\times 28$ grid, with output channel counts $64$, $128$, $32$ and $32$. Joining channels gives a $28\times 28\times 256$ output becauseThe equal spatial grid sizes are an assumption ensured by the branch design. Channel joining does not add corresponding values into one channel; it retains the distinct branch outputs as separate channels of the combined neural network feature map.

A cost example explains the reduction branch. Ignoring biases, a dense three-by-three convolutional layer from $192$ input channels to $128$ output channels hasweights. Reducing first from $192$ to $32$ channels with a one-by-one pointwise convolution, then applying the three-by-three layer, usesweights. This own example isolates channel dimensions; it is not a reported layer configuration or a claim that the two computations represent identical functions.

The original paper presents this design in the GoogLeNet network and evaluates image classification and detection. Later Rethinking the Inception Architecture for Computer Vision by Christian Szegedy and others develops additional factorization and training choices. The family name therefore does not identify one immutable list of layers or one universal performance result.^[ref][ref]

A pointwise convolution uses a spatial kernel of size one to combine neural network channels independently at each spatial position. For a two-dimensional input this is a one-by-one convolutional layer. In the ordinary dense-channel, unit-stride case, the same affine map acts on the channel vector at every grid point.^[ref]

Let $x_{i,j}\in {\mathbb{R}}^{C_{\mathrm{in}}}$ be the channel vector at position $(i,j)$. With weight matrix $M$ and neural network bias vector $b$, the output isThis is matrix multiplication across channels, not an average of neighboring spatial positions. The same $M$ and $b$ are reused throughout the grid.

For an example without biases, takeAt a point with input $(1,2,3{)}^{\mathsf{T}}$ the output is $(5,1{)}^{\mathsf{T}}$. At another point with input $(3,0,4{)}^{\mathsf{T}}$ it is $(3,4{)}^{\mathsf{T}}$. A three-channel neural network feature map has become a two-channel map without mixing the two positions.

There are $C_{\mathrm{in}}{C}_{\mathrm{out}}$ weights, plus $C_{\mathrm{out}}$ optional biases. Thus three input and two output channels require six weights, or eight parameters including biases. A dense three-by-three filter with the same channel counts requires $9(3)(2)=54$ weights, or fifty-six parameters with biases. This comparison counts parameters for the stated connectivity; it does not claim that the two layers compute equivalent functions.

Channel reduction can make a later spatial convolutional layer cheaper. The original Inception uses one-by-one reductions before some larger filters. A pointwise operation can also increase the channel count; size one refers to its spatial footprint, not to the number of channels.^[ref] With no intervening spatial operation, pointwise layers and pointwise activation functions alone do not enlarge the spatial neural network receptive field.