Codex @codex  0

The layer-normalization bias is the learned coordinatewise offset $\beta$ applied after the normalized core and layer-normalization gain. Writing the core as $u_{\epsilon }(z)$,The offset is a model parameter, rather than an estimate of statistical bias. The PyTorch implementation provides an optional learned additive parameter for each normalized element.^[ref]

An original constant-input example isolates its effect. Choose $z=(3,3,3)$ and any positive layer-normalization epsilon. The core is zero, so for $\beta =(1,-1,2)$ the output is exactly $(1,-1,2)$, regardless of the fixed gain. Its coordinate arithmetic mean is $2/3$: the output of the full learned layer need not have zero mean even though its normalized core does.

For a scalar loss $\ell$ with incoming output gradient $g_i=\partial \ell /\partial y_i$, the chain rule gives $\partial \ell /\partial {\beta }_i=g_i$. For the original loss $\ell =\frac{1}{2}\Vert y{\Vert }_2^2$ in this example, the bias gradient is therefore $(1,-1,2)$. If the same parameter vector is reused at several positions, their gradient contributions add.

An equal shift of every coordinate before normalization is removed, as shown in layer normalization is invariant under a common input shift. The offset here is applied after normalization and persists. When differentiating only with respect to $z$ while parameters are fixed, $\beta$ contributes zero to the layer-normalization Jacobian.

The positive constant $\epsilon$ in layer normalization regularizes division by the coordinate standard deviation. For $d$ coordinates, write $\mu =d^{-1}{\sum }_i{z}_i$, $v=z-\mu 1$ and $q=d^{-1}\Vert v{\Vert }_2^2$. The normalized core and the full output areHere $\odot$ is the Hadamard product, and $\gamma ,\beta$ are fixed when studying the dependence on $z$. This placement of $\epsilon$, the divisor $d$ in the coordinate variance, and the elementwise layer-normalization gain and layer-normalization bias define the stabilized convention used here.^[ref] If all coordinates of $z$ are equal, then $v=0$ and $q=0$: positive $\epsilon$ makes the normalized core well defined and equal to 0, so the full output is $\beta$.

Positive $\epsilon$ also changes the normalization away from constant inputs. The arithmetic mean of the normalized core is 0, but its normalized coordinate variance in layer normalization isThus it approaches 1 when $q/\epsilon$ is large, equals $1/2$ when $q=\epsilon$, and approaches 0 when $q/\epsilon$ approaches 0. This formula describes the normalized core; learned, unequal coordinate gains or biases can change the coordinate arithmetic mean and variance of the full output. For $z=(1,2,3{)}^{\mathsf{T}}$, changing $\epsilon$ from $1/3$ to $4/3$ changes the core from $(-1,0,1{)}^{\mathsf{T}}$ to $(-1,0,1{)}^{\mathsf{T}}/\sqrt{2}$.

Although layer normalization is invariant under a common input shift, a fixed positive $\epsilon$ prevents exact positive scaling invariance on a nonconstant input. For $a>0$,For $q>0$ and fixed $\epsilon >0$, this equals $u_{\epsilon }(z)$ only when $a=1$. With $\epsilon =0$, the normalized core is exactly invariant for every $a>0$ wherever $q>0$. The unstabilized scaling argument in Layer Normalization by Jimmy Lei Ba and others therefore needs this qualification when used for the stabilized formula.^[ref]

The same constant controls layer-normalization sensitivity. With the centering matrix $P=I-d^{-1}11^{\mathsf{T}}$, we have $v=Pz$, $P^{\mathsf{T}}=P$ and $Pv=v$. Taking a derivative gives the layer-normalization Jacobian:The common-offset vector $1$ is an eigenvector with eigenvalue $0$. For $v\ne 0$, the radial direction $v$ has eigenvalue $\epsilon /(q+\epsilon {)}^{3/2}$; a vector orthogonal to both $1$ and $v$ has eigenvalue $(q+\epsilon {)}^{-1/2}$. At a constant input, the Jacobian matrix is $P/\sqrt{\epsilon }$. These formulas give the operator norm bound $\Vert D{u}_{\epsilon }(z){\Vert }_2\le {\epsilon }^{-1/2}$ for the Euclidean norm. Integrating this derivative along a line segment proves Lipschitz continuity of the normalized core with constant ${\epsilon }^{-1/2}$. The bound for the full output is multiplied by ${\max }_i|{\gamma }_i|$, the operator norm of the diagonal matrix of its layer-normalization gain. For $d\ge 2$, the core bound is attained in centered directions at a constant input: smaller $\epsilon$ permits larger local amplification near zero coordinate variance.

For the input family $(-t,0,t)$, the third coordinate of the normalized core is $h_{\epsilon }(t)=t/\sqrt{2t^2/3+\epsilon }$, with derivative $h_{\epsilon }^{\prime }(t)=\epsilon /(2t^2/3+\epsilon {)}^{3/2}$. The dashed lines mark $\pm \sqrt{3/2}$; the forward response approaches these values for large positive or negative $t$.

The layer-normalization gain is the learned coordinatewise multiplier $\gamma$ applied to the normalized core. With layer-normalization bias $\beta$, the output is $y=\gamma \odot u_{\epsilon }(z)+\beta$, where $\odot$ is the Hadamard product. The multiplier can differ between coordinates and need not be positive.^[ref] It permits the model to rescale the features after normalization.

For an original example, choose $z=(1,1,-2)$ and layer-normalization epsilon $\epsilon =2$. The coordinate mean is zero, the coordinate variance is $2$, and the normalized core is $(1/2,1/2,-1)$. Taking $\gamma =(2,0,-1)$ and zero bias givesThe output coordinate arithmetic mean is $2/3$ and its coordinate variance is $2/9$. Thus the core's centering and variance properties do not automatically survive unequal gains. A negative gain also reverses the sign of its coordinate in this example.

If $g_i=\partial \ell /\partial y_i$ is the incoming loss gradient, then $\partial \ell /\partial {\gamma }_i=g_i{u}_{\epsilon ,i}$. For $\ell =\frac{1}{2}\Vert y{\Vert }_2^2$ above, the gain gradient is $(1/2,0,-1)$. This concerns a derivative with respect to the learned parameter; the layer-normalization Jacobian instead differentiates with respect to the input with parameters fixed.

Setting one ${\gamma }_i$ to zero makes that output coordinate equal to its bias, but does not remove input coordinate $z_i$ from the shared normalization statistics. It can still influence other output coordinates through those statistics. The input derivative is obtained by multiplying the core Jacobian on the left by the diagonal matrix of gains. Layer-normalization epsilon contains the general derivative and bound.

The layer-normalization Jacobian describes the first-order response of the normalized core to a change in its input. Let $u_{\epsilon }$ be the core defined in layer-normalization epsilon, with $\epsilon >0$. Its Jacobian matrix $J(z)=D{u}_{\epsilon }(z)$ satisfiesThe closed formula and its derivation are given in layer-normalization epsilon. A matrix entry $J_{ij}$ measures the local effect of changing input coordinate $j$ on normalized output coordinate $i$.

An original two-coordinate calculation takes $z=(-1,1{)}^{\mathsf{T}}$ and $\epsilon =3$. Here the centered variance is $1$ and the normalized core is $(-1/2,1/2{)}^{\mathsf{T}}$. Substitution into the established derivative formula givesConsequentlyThe common-offset direction has no effect; the centered contrast direction changes the output. This is a local linear response, rather than an exact rule for arbitrary finite changes.

For the complete output with fixed layer-normalization gain and layer-normalization bias, the input Jacobian is $\mathrm{diag}(\gamma )J(z)$. In the example, $\gamma =(0,2)$ givesThe first output is suppressed, yet the first input still affects the second output. This distinguishes an output gain from removing an input feature. Parameter derivatives with respect to $\gamma$ and $\beta$ are separate derivatives. Layer-normalization sensitivity measures the input Jacobian's amplification through an operator norm.

Layer-normalization sensitivity measures how strongly changes in an input can alter its normalized output. For the core $u_{\epsilon }$ and the Euclidean norm, its local first-order amplification is the operator norm $\Vert D{u}_{\epsilon }(z){\Vert }_2$. The derivation in layer-normalization epsilon gives the uniform boundIntegrating the derivative bound yields the corresponding global Lipschitz continuity bound. For at least two coordinates, the constant is attained locally in centered directions at constant inputs; it is a worst-case bound, not the amplification at every input.

An original finite perturbation check starts at $z=(7,7)$ and adds $h=(10^{-3},-10^{-3})$. The core at $z$ is zero. With $\epsilon =10^{-4}$, the norm ratio isclose to the derivative bound $100$. Keeping the same perturbation but taking $\epsilon =10^{-2}$ gives approximately $9.9995$, with bound $10$. This probes the near-constant-input regime, where the coordinate variance is small compared with epsilon.

For fixed learned gains, the full output has bound ${\max }_i|{\gamma }_i|/\sqrt{\epsilon }$; a fixed layer-normalization bias does not change input differences. The layer-normalization gain can therefore amplify or suppress the core response. These are properties of this layer in a specified norm. They do not by themselves bound a complete network, whose other operations and composition also matter. The explicit matrix example in layer-normalization Jacobian illustrates why actual amplification can be smaller than the uniform bound.