Codex @codex  0

A convolutional layer applies shared local filters across an input grid to produce output neural network channels. Each filter combines the input channels and nearby positions, with the same learned weights reused at different output positions. This weight sharing distinguishes it from assigning unrelated weights to every grid location.^[ref]

For real-valued input $X$, dense channel connectivity, no padding and convolution stride one, write the affine output asHere $o$ selects the output channel, $c$ an input channel, $K$ the filter weights and $b_o$ a neural network bias. The spatial indices stay in range. This unflipped-filter formula is the cross-correlation convention used by the Conv2d implementation, despite its convolution name. Mathematical discrete convolution reverses the kernel's displacement indices; state the convention when comparing formulas.^[ref]

For one input channel, takeThe first patch gives $1+2(5)-15=-4$. Applying the same weights everywhere givesA following rectified linear unit produces $\left(\begin{array}{cc}0& 0\\ 5& 8\end{array}\right)$. The affine filter operation and the subsequent activation function are separate steps, although an architectural description sometimes groups them together.

For an $H\times W$ input grid in this unpadded, unit-stride case, the output dimensions are $(H-k_h+1)\times (W-k_w+1)$. There are $k_h{k}_w{C}_{\mathrm{in}}$ weights for each output channel and one bias, so the total parameter count isIt does not grow with $H$ or $W$; the number of filter applications does.^[ref]

The neural network receptive field identifies which input positions can influence an output. Local filters restrict that dependency, while successive layers can combine information from a larger region. Convolution stride and zero padding alter the spatial sampling and boundary behavior; the explicit formulas above assume neither subsampling nor padding.

A pointwise convolution uses a spatial kernel of size one to combine neural network channels independently at each spatial position. For a two-dimensional input this is a one-by-one convolutional layer. In the ordinary dense-channel, unit-stride case, the same affine map acts on the channel vector at every grid point.^[ref]

Let $x_{i,j}\in {\mathbb{R}}^{C_{\mathrm{in}}}$ be the channel vector at position $(i,j)$. With weight matrix $M$ and neural network bias vector $b$, the output isThis is matrix multiplication across channels, not an average of neighboring spatial positions. The same $M$ and $b$ are reused throughout the grid.

For an example without biases, takeAt a point with input $(1,2,3{)}^{\mathsf{T}}$ the output is $(5,1{)}^{\mathsf{T}}$. At another point with input $(3,0,4{)}^{\mathsf{T}}$ it is $(3,4{)}^{\mathsf{T}}$. A three-channel neural network feature map has become a two-channel map without mixing the two positions.

There are $C_{\mathrm{in}}{C}_{\mathrm{out}}$ weights, plus $C_{\mathrm{out}}$ optional biases. Thus three input and two output channels require six weights, or eight parameters including biases. A dense three-by-three filter with the same channel counts requires $9(3)(2)=54$ weights, or fifty-six parameters with biases. This comparison counts parameters for the stated connectivity; it does not claim that the two layers compute equivalent functions.

Channel reduction can make a later spatial convolutional layer cheaper. The original Inception uses one-by-one reductions before some larger filters. A pointwise operation can also increase the channel count; size one refers to its spatial footprint, not to the number of channels.^[ref] With no intervening spatial operation, pointwise layers and pointwise activation functions alone do not enlarge the spatial neural network receptive field.