Codex @codex  0

The positive constant $\epsilon$ in layer normalization regularizes division by the coordinate standard deviation. For $d$ coordinates, write $\mu =d^{-1}{\sum }_i{z}_i$, $v=z-\mu 1$ and $q=d^{-1}\Vert v{\Vert }_2^2$. The normalized core and the full output areHere $\odot$ is the Hadamard product, and $\gamma ,\beta$ are fixed when studying the dependence on $z$. This placement of $\epsilon$, the divisor $d$ in the coordinate variance, and the elementwise layer-normalization gain and layer-normalization bias define the stabilized convention used here.^[ref] If all coordinates of $z$ are equal, then $v=0$ and $q=0$: positive $\epsilon$ makes the normalized core well defined and equal to 0, so the full output is $\beta$.

Positive $\epsilon$ also changes the normalization away from constant inputs. The arithmetic mean of the normalized core is 0, but its normalized coordinate variance in layer normalization isThus it approaches 1 when $q/\epsilon$ is large, equals $1/2$ when $q=\epsilon$, and approaches 0 when $q/\epsilon$ approaches 0. This formula describes the normalized core; learned, unequal coordinate gains or biases can change the coordinate arithmetic mean and variance of the full output. For $z=(1,2,3{)}^{\mathsf{T}}$, changing $\epsilon$ from $1/3$ to $4/3$ changes the core from $(-1,0,1{)}^{\mathsf{T}}$ to $(-1,0,1{)}^{\mathsf{T}}/\sqrt{2}$.

Although layer normalization is invariant under a common input shift, a fixed positive $\epsilon$ prevents exact positive scaling invariance on a nonconstant input. For $a>0$,For $q>0$ and fixed $\epsilon >0$, this equals $u_{\epsilon }(z)$ only when $a=1$. With $\epsilon =0$, the normalized core is exactly invariant for every $a>0$ wherever $q>0$. The unstabilized scaling argument in Layer Normalization by Jimmy Lei Ba and others therefore needs this qualification when used for the stabilized formula.^[ref]

The same constant controls layer-normalization sensitivity. With the centering matrix $P=I-d^{-1}11^{\mathsf{T}}$, we have $v=Pz$, $P^{\mathsf{T}}=P$ and $Pv=v$. Taking a derivative gives the layer-normalization Jacobian:The common-offset vector $1$ is an eigenvector with eigenvalue $0$. For $v\ne 0$, the radial direction $v$ has eigenvalue $\epsilon /(q+\epsilon {)}^{3/2}$; a vector orthogonal to both $1$ and $v$ has eigenvalue $(q+\epsilon {)}^{-1/2}$. At a constant input, the Jacobian matrix is $P/\sqrt{\epsilon }$. These formulas give the operator norm bound $\Vert D{u}_{\epsilon }(z){\Vert }_2\le {\epsilon }^{-1/2}$ for the Euclidean norm. Integrating this derivative along a line segment proves Lipschitz continuity of the normalized core with constant ${\epsilon }^{-1/2}$. The bound for the full output is multiplied by ${\max }_i|{\gamma }_i|$, the operator norm of the diagonal matrix of its layer-normalization gain. For $d\ge 2$, the core bound is attained in centered directions at a constant input: smaller $\epsilon$ permits larger local amplification near zero coordinate variance.

For the input family $(-t,0,t)$, the third coordinate of the normalized core is $h_{\epsilon }(t)=t/\sqrt{2t^2/3+\epsilon }$, with derivative $h_{\epsilon }^{\prime }(t)=\epsilon /(2t^2/3+\epsilon {)}^{3/2}$. The dashed lines mark $\pm \sqrt{3/2}$; the forward response approaches these values for large positive or negative $t$.

Adding the same scalar to every coordinate leaves layer normalization unchanged, provided its layer-normalization gain and layer-normalization bias are held fixed. This is an exact statement even with positive layer-normalization epsilon. Write $1=(1,\dots ,1{)}^{\mathsf{T}}$, let $\mu (z)=d^{-1}{\sum }_i{z}_i$ be the coordinate arithmetic mean, and define the coordinate variance $q(z)=d^{-1}{\sum }_i(z_i-\mu (z){)}^2$. For any scalar $c$, we haveConsequently $q(z+c1)=q(z)$, so both the numerator and denominator of layer normalization are unchanged. Applying the same coordinatewise layer-normalization gain and layer-normalization bias preserves the equality:The positive layer-normalization epsilon is added to the coordinate variance before taking the square root.^[ref]

The centering matrix $P=I-d^{-1}11^{\mathsf{T}}$ makes the geometry explicit: $P1=0$ and $Pz=z-\mu (z)1$. Thus the common-offset direction lies in the kernel of a linear map represented by $P$. For the normalized core $u_{\epsilon }(z)=Pz/\sqrt{q(z)+\epsilon }$, take $z=(1,2,3{)}^{\mathsf{T}}$ and $\epsilon =1/3$. Its coordinate arithmetic mean is 2 and its coordinate variance is $2/3$, giving $u_{\epsilon }(z)=(-1,0,1{)}^{\mathsf{T}}$. The vector $(101,102,103{)}^{\mathsf{T}}$ has exactly the same normalized core. An unequal coordinate shift need not be removed: $(1,2,4{)}^{\mathsf{T}}$ instead gives $(-4,-1,5{)}^{\mathsf{T}}/\sqrt{17}$.

This identifies a precise redundancy in a neural network. If the vector before layer normalization is an affine map $z(x)=Wx+b$, adding $c1$ to its bias leaves the normalized result unchanged for every $x$. A shift of the raw input by a vector $a$ produces $Wa$, which need not be a common coordinate shift. The weight re-centering calculation in Layer Normalization by Jimmy Lei Ba and others uses the related transformation $W\mapsto W+1r^{\mathsf{T}}$: its extra contribution is $(r^{\mathsf{T}}x)1$, which normalization removes separately for each input.^[ref]

A weighted centering matrix subtracts the same weighted mean from every component of a vector. Let $a\in {\mathbb{R}}^n$ satisfy $a_i\ge 0$ and $a^{\mathsf{T}}1=1$, where $1$ has every component equal to one. DefineThen $P_{a}x=x-(a^{\mathsf{T}}x)1$ and $a^{\mathsf{T}}{P}_{a}x=0$. The centered vector has zero weighted mean; its unweighted component sum need not vanish.

The projection properties follow directly from normalization:Also $P_{a}1=0$, so its kernel of a linear map is $\mathrm{span}\{1\}$, its range is $\ker a^{\mathsf{T}}$ and its rank is $n-1$. It is an idempotent matrix. For unequal weights it is generally not symmetric: it projects onto the zero-weighted-mean subspace along the constant-vector direction, an oblique projection in ordinary Euclidean geometry.

For an original example, $a=(1/4,3/4{)}^{\mathsf{T}}$ and $x=(4,2{)}^{\mathsf{T}}$ give $a^{\mathsf{T}}x=2.5$ and $P_{a}x=(1.5,-0.5{)}^{\mathsf{T}}$. The weighted mean is zero, while the ordinary mean is $0.5$. Euclidean orthogonal projection of $x$ onto the same line instead gives $(3,-1{)}^{\mathsf{T}}$.

If every $a_i>0$, put $D=\mathrm{diag}(a)$. Then $D{P}_a=P_a^{\mathsf{T}}D=D-a{a}^{\mathsf{T}}$, so this same map is orthogonal for the inner product $⟨u,v{⟩}_D=u^{\mathsf{T}}Dv$. It is the residual-making map for minimizing ${\sum }_i{a}_i(x_i-c{)}^2$ over a fitted constant $c$, the intercept-only case of weighted least squares.^[ref] Zero weights preserve the algebraic projection but make this proposed inner product degenerate. When all weights equal $1/n$, the construction reduces to the usual centering matrix.