= Layer normalization is invariant under a common input shift
Adding the same <scalar> to every coordinate leaves <layer normalization> unchanged, provided its <layer-normalization gain> and <layer-normalization bias> are held fixed. This is an exact statement even with positive <layer-normalization epsilon>. Write $\mathbf1=(1,\ldots,1)^{\mathsf T}$, let $\mu(z)=d^{-1}\sum_i z_i$ be the coordinate <arithmetic mean>, and define the coordinate <variance> $q(z)=d^{-1}\sum_i(z_i-\mu(z))^2$. For any <scalar> $c$, we have
$$
\mu(z+c\mathbf1)=\mu(z)+c,\qquad
z+c\mathbf1-\mu(z+c\mathbf1)\mathbf1=z-\mu(z)\mathbf1.
$$
Consequently $q(z+c\mathbf1)=q(z)$, so both the numerator and denominator of <layer normalization> are unchanged. Applying the same coordinatewise <layer-normalization gain> and <layer-normalization bias> preserves the equality:
$$
\boxed{\operatorname{LN}_\varepsilon(z+c\mathbf1)=\operatorname{LN}_\varepsilon(z).}
$$
The positive <layer-normalization epsilon> is added to the coordinate <variance> before taking the <square root>.<LayerNorm in the PyTorch documentation>[Equation, correction=0 convention and Parameters/eps]{ref}
The <centering matrix> $P=I-d^{-1}\mathbf1\mathbf1^{\mathsf T}$ makes the geometry explicit: $P\mathbf1=0$ and $Pz=z-\mu(z)\mathbf1$. Thus the common-offset direction lies in the <kernel of a linear map> represented by $P$. For the normalized core $u_\varepsilon(z)=Pz/\sqrt{q(z)+\varepsilon}$, take $z=(1,2,3)^{\mathsf T}$ and $\varepsilon=1/3$. Its coordinate <arithmetic mean> is 2 and its coordinate <variance> is $2/3$, giving $u_\varepsilon(z)=(-1,0,1)^{\mathsf T}$. The <vector> $(101,102,103)^{\mathsf T}$ has exactly the same normalized core. An unequal coordinate shift need not be removed: $(1,2,4)^{\mathsf T}$ instead gives $(-4,-1,5)^{\mathsf T}/\sqrt{17}$.
This identifies a precise redundancy in a <neural network>. If the <vector> before <layer normalization> is an <affine map> $z(x)=Wx+b$, adding $c\mathbf1$ to its bias leaves the normalized result unchanged for every $x$. A shift of the raw input by a <vector> $a$ produces $Wa$, which need not be a common coordinate shift. The weight re-centering calculation in <Layer Normalization by Jimmy Lei Ba and others> uses the related transformation $W\mapsto W+\mathbf1 r^{\mathsf T}$: its extra contribution is $(r^{\mathsf T}x)\mathbf1$, which normalization removes separately for each input.<Layer Normalization by Jimmy Lei Ba and others>[§5.1, equation 6: common shifts produced by weight-matrix re-centering]{ref}
Back to article page
