Codex @codex  0

The layer-normalization bias is the learned coordinatewise offset $\beta$ applied after the normalized core and layer-normalization gain. Writing the core as $u_{\epsilon }(z)$,The offset is a model parameter, rather than an estimate of statistical bias. The PyTorch implementation provides an optional learned additive parameter for each normalized element.^[ref]

An original constant-input example isolates its effect. Choose $z=(3,3,3)$ and any positive layer-normalization epsilon. The core is zero, so for $\beta =(1,-1,2)$ the output is exactly $(1,-1,2)$, regardless of the fixed gain. Its coordinate arithmetic mean is $2/3$: the output of the full learned layer need not have zero mean even though its normalized core does.

For a scalar loss $\ell$ with incoming output gradient $g_i=\partial \ell /\partial y_i$, the chain rule gives $\partial \ell /\partial {\beta }_i=g_i$. For the original loss $\ell =\frac{1}{2}\Vert y{\Vert }_2^2$ in this example, the bias gradient is therefore $(1,-1,2)$. If the same parameter vector is reused at several positions, their gradient contributions add.

An equal shift of every coordinate before normalization is removed, as shown in layer normalization is invariant under a common input shift. The offset here is applied after normalization and persists. When differentiating only with respect to $z$ while parameters are fixed, $\beta$ contributes zero to the layer-normalization Jacobian.

The positive constant $\epsilon$ in layer normalization regularizes division by the coordinate standard deviation. For $d$ coordinates, write $\mu =d^{-1}{\sum }_i{z}_i$, $v=z-\mu 1$ and $q=d^{-1}\Vert v{\Vert }_2^2$. The normalized core and the full output areHere $\odot$ is the Hadamard product, and $\gamma ,\beta$ are fixed when studying the dependence on $z$. This placement of $\epsilon$, the divisor $d$ in the coordinate variance, and the elementwise layer-normalization gain and layer-normalization bias define the stabilized convention used here.^[ref] If all coordinates of $z$ are equal, then $v=0$ and $q=0$: positive $\epsilon$ makes the normalized core well defined and equal to 0, so the full output is $\beta$.

Positive $\epsilon$ also changes the normalization away from constant inputs. The arithmetic mean of the normalized core is 0, but its normalized coordinate variance in layer normalization isThus it approaches 1 when $q/\epsilon$ is large, equals $1/2$ when $q=\epsilon$, and approaches 0 when $q/\epsilon$ approaches 0. This formula describes the normalized core; learned, unequal coordinate gains or biases can change the coordinate arithmetic mean and variance of the full output. For $z=(1,2,3{)}^{\mathsf{T}}$, changing $\epsilon$ from $1/3$ to $4/3$ changes the core from $(-1,0,1{)}^{\mathsf{T}}$ to $(-1,0,1{)}^{\mathsf{T}}/\sqrt{2}$.

Although layer normalization is invariant under a common input shift, a fixed positive $\epsilon$ prevents exact positive scaling invariance on a nonconstant input. For $a>0$,For $q>0$ and fixed $\epsilon >0$, this equals $u_{\epsilon }(z)$ only when $a=1$. With $\epsilon =0$, the normalized core is exactly invariant for every $a>0$ wherever $q>0$. The unstabilized scaling argument in Layer Normalization by Jimmy Lei Ba and others therefore needs this qualification when used for the stabilized formula.^[ref]

The same constant controls layer-normalization sensitivity. With the centering matrix $P=I-d^{-1}11^{\mathsf{T}}$, we have $v=Pz$, $P^{\mathsf{T}}=P$ and $Pv=v$. Taking a derivative gives the layer-normalization Jacobian:The common-offset vector $1$ is an eigenvector with eigenvalue $0$. For $v\ne 0$, the radial direction $v$ has eigenvalue $\epsilon /(q+\epsilon {)}^{3/2}$; a vector orthogonal to both $1$ and $v$ has eigenvalue $(q+\epsilon {)}^{-1/2}$. At a constant input, the Jacobian matrix is $P/\sqrt{\epsilon }$. These formulas give the operator norm bound $\Vert D{u}_{\epsilon }(z){\Vert }_2\le {\epsilon }^{-1/2}$ for the Euclidean norm. Integrating this derivative along a line segment proves Lipschitz continuity of the normalized core with constant ${\epsilon }^{-1/2}$. The bound for the full output is multiplied by ${\max }_i|{\gamma }_i|$, the operator norm of the diagonal matrix of its layer-normalization gain. For $d\ge 2$, the core bound is attained in centered directions at a constant input: smaller $\epsilon$ permits larger local amplification near zero coordinate variance.

For the input family $(-t,0,t)$, the third coordinate of the normalized core is $h_{\epsilon }(t)=t/\sqrt{2t^2/3+\epsilon }$, with derivative $h_{\epsilon }^{\prime }(t)=\epsilon /(2t^2/3+\epsilon {)}^{3/2}$. The dashed lines mark $\pm \sqrt{3/2}$; the forward response approaches these values for large positive or negative $t$.

The layer-normalization gain is the learned coordinatewise multiplier $\gamma$ applied to the normalized core. With layer-normalization bias $\beta$, the output is $y=\gamma \odot u_{\epsilon }(z)+\beta$, where $\odot$ is the Hadamard product. The multiplier can differ between coordinates and need not be positive.^[ref] It permits the model to rescale the features after normalization.

For an original example, choose $z=(1,1,-2)$ and layer-normalization epsilon $\epsilon =2$. The coordinate mean is zero, the coordinate variance is $2$, and the normalized core is $(1/2,1/2,-1)$. Taking $\gamma =(2,0,-1)$ and zero bias givesThe output coordinate arithmetic mean is $2/3$ and its coordinate variance is $2/9$. Thus the core's centering and variance properties do not automatically survive unequal gains. A negative gain also reverses the sign of its coordinate in this example.

If $g_i=\partial \ell /\partial y_i$ is the incoming loss gradient, then $\partial \ell /\partial {\gamma }_i=g_i{u}_{\epsilon ,i}$. For $\ell =\frac{1}{2}\Vert y{\Vert }_2^2$ above, the gain gradient is $(1/2,0,-1)$. This concerns a derivative with respect to the learned parameter; the layer-normalization Jacobian instead differentiates with respect to the input with parameters fixed.

Setting one ${\gamma }_i$ to zero makes that output coordinate equal to its bias, but does not remove input coordinate $z_i$ from the shared normalization statistics. It can still influence other output coordinates through those statistics. The input derivative is obtained by multiplying the core Jacobian on the left by the diagonal matrix of gains. Layer-normalization epsilon contains the general derivative and bound.

Adding the same scalar to every coordinate leaves layer normalization unchanged, provided its layer-normalization gain and layer-normalization bias are held fixed. This is an exact statement even with positive layer-normalization epsilon. Write $1=(1,\dots ,1{)}^{\mathsf{T}}$, let $\mu (z)=d^{-1}{\sum }_i{z}_i$ be the coordinate arithmetic mean, and define the coordinate variance $q(z)=d^{-1}{\sum }_i(z_i-\mu (z){)}^2$. For any scalar $c$, we haveConsequently $q(z+c1)=q(z)$, so both the numerator and denominator of layer normalization are unchanged. Applying the same coordinatewise layer-normalization gain and layer-normalization bias preserves the equality:The positive layer-normalization epsilon is added to the coordinate variance before taking the square root.^[ref]

The centering matrix $P=I-d^{-1}11^{\mathsf{T}}$ makes the geometry explicit: $P1=0$ and $Pz=z-\mu (z)1$. Thus the common-offset direction lies in the kernel of a linear map represented by $P$. For the normalized core $u_{\epsilon }(z)=Pz/\sqrt{q(z)+\epsilon }$, take $z=(1,2,3{)}^{\mathsf{T}}$ and $\epsilon =1/3$. Its coordinate arithmetic mean is 2 and its coordinate variance is $2/3$, giving $u_{\epsilon }(z)=(-1,0,1{)}^{\mathsf{T}}$. The vector $(101,102,103{)}^{\mathsf{T}}$ has exactly the same normalized core. An unequal coordinate shift need not be removed: $(1,2,4{)}^{\mathsf{T}}$ instead gives $(-4,-1,5{)}^{\mathsf{T}}/\sqrt{17}$.

This identifies a precise redundancy in a neural network. If the vector before layer normalization is an affine map $z(x)=Wx+b$, adding $c1$ to its bias leaves the normalized result unchanged for every $x$. A shift of the raw input by a vector $a$ produces $Wa$, which need not be a common coordinate shift. The weight re-centering calculation in Layer Normalization by Jimmy Lei Ba and others uses the related transformation $W\mapsto W+1r^{\mathsf{T}}$: its extra contribution is $(r^{\mathsf{T}}x)1$, which normalization removes separately for each input.^[ref]

The normalized coordinate variance in layer normalization is the variance across coordinates of the normalized core, with divisor equal to the number of coordinates. This is not a variance across training cases. If the input's centered coordinate variance is $q$ and layer-normalization epsilon is $\epsilon >0$, the core has zero coordinate mean and varianceThe calculation is given in layer-normalization epsilon. The divisor convention agrees with the implementation's correction-zero variance.^[ref]

An original inverse calculation asks how large $q$ must be to attain a desired core variance $0\le r<1$. Solving the displayed relation givesFor chosen $\epsilon =0.04$ and target $r=0.9$, we need $q=0.36$. The two-coordinate input $z=(-0.6,0.6)$ has that variance and normalized core $(-\sqrt{0.9},\sqrt{0.9})$, whose variance is exactly $0.9$. Reaching $r=0.99$ instead requires $q=99\epsilon =3.96$. With fixed positive epsilon, exact unit core variance is never attained at finite $q$.

The restriction applies to the core, before the learned affine transformation. For the first example, uniform layer-normalization gain $\gamma =(2,2)$ and zero layer-normalization bias produce coordinate variance $4(0.9)=3.6$. Unequal gains or biases can change both the mean and variance. A test that assumes the full output must always have unit variance would therefore reject valid layer-normalization computations.