 Layer normalization is invariant under a common input shift

Adding the same scalar to every coordinate leaves layer normalization unchanged, provided its layer-normalization gain and layer-normalization bias are held fixed. This is an exact statement even with positive layer-normalization epsilon. Write $1=(1,\dots ,1{)}^{\mathsf{T}}$, let $\mu (z)=d^{-1}{\sum }_i{z}_i$ be the coordinate arithmetic mean, and define the coordinate variance $q(z)=d^{-1}{\sum }_i(z_i-\mu (z){)}^2$. For any scalar $c$, we haveConsequently $q(z+c1)=q(z)$, so both the numerator and denominator of layer normalization are unchanged. Applying the same coordinatewise layer-normalization gain and layer-normalization bias preserves the equality:The positive layer-normalization epsilon is added to the coordinate variance before taking the square root.^[ref]

The centering matrix $P=I-d^{-1}11^{\mathsf{T}}$ makes the geometry explicit: $P1=0$ and $Pz=z-\mu (z)1$. Thus the common-offset direction lies in the kernel of a linear map represented by $P$. For the normalized core $u_{\epsilon }(z)=Pz/\sqrt{q(z)+\epsilon }$, take $z=(1,2,3{)}^{\mathsf{T}}$ and $\epsilon =1/3$. Its coordinate arithmetic mean is 2 and its coordinate variance is $2/3$, giving $u_{\epsilon }(z)=(-1,0,1{)}^{\mathsf{T}}$. The vector $(101,102,103{)}^{\mathsf{T}}$ has exactly the same normalized core. An unequal coordinate shift need not be removed: $(1,2,4{)}^{\mathsf{T}}$ instead gives $(-4,-1,5{)}^{\mathsf{T}}/\sqrt{17}$.

This identifies a precise redundancy in a neural network. If the vector before layer normalization is an affine map $z(x)=Wx+b$, adding $c1$ to its bias leaves the normalized result unchanged for every $x$. A shift of the raw input by a vector $a$ produces $Wa$, which need not be a common coordinate shift. The weight re-centering calculation in Layer Normalization by Jimmy Lei Ba and others uses the related transformation $W\mapsto W+1r^{\mathsf{T}}$: its extra contribution is $(r^{\mathsf{T}}x)1$, which normalization removes separately for each input.^[ref]