= Layer-normalization epsilon
{title2=$\varepsilon$}
The positive constant $\varepsilon$ in <layer normalization> regularizes division by the coordinate <standard deviation>. For $d$ coordinates, write $\mu=d^{-1}\sum_i z_i$, $v=z-\mu\mathbf1$ and $q=d^{-1}\|v\|_2^2$. The normalized core and the full output are
$$
u_\varepsilon(z)=\frac{v}{\sqrt{q+\varepsilon}},\qquad
\operatorname{LN}_\varepsilon(z)=\gamma\odot u_\varepsilon(z)+\beta.
$$
Here $\odot$ is the <Hadamard product>, and $\gamma,\beta$ are fixed when studying the dependence on $z$. This placement of $\varepsilon$, the divisor $d$ in the coordinate <variance>, and the elementwise <layer-normalization gain> and <layer-normalization bias> define the stabilized convention used here.<LayerNorm in the PyTorch documentation>[Equation; correction=0; Parameters/eps and elementwise_affine]{ref} If all coordinates of $z$ are equal, then $v=0$ and $q=0$: positive $\varepsilon$ makes the normalized core well defined and equal to 0, so the full output is $\beta$.
Positive $\varepsilon$ also changes the normalization away from constant inputs. The <arithmetic mean> of the normalized core is 0, but its <normalized coordinate variance in layer normalization> is
$$
\boxed{\frac1d\sum_i u_{\varepsilon,i}(z)^2=\frac{q}{q+\varepsilon}.}
$$
Thus it approaches 1 when $q/\varepsilon$ is large, equals $1/2$ when $q=\varepsilon$, and approaches 0 when $q/\varepsilon$ approaches 0. This formula describes the normalized core; learned, unequal coordinate gains or biases can change the coordinate <arithmetic mean> and <variance> of the full output. For $z=(1,2,3)^{\mathsf T}$, changing $\varepsilon$ from $1/3$ to $4/3$ changes the core from $(-1,0,1)^{\mathsf T}$ to $(-1,0,1)^{\mathsf T}/\sqrt2$.
Although <layer normalization is invariant under a common input shift>, a fixed positive $\varepsilon$ prevents exact positive scaling invariance on a nonconstant input. For $a>0$,
$$
u_\varepsilon(az)=\frac{a v}{\sqrt{a^2q+\varepsilon}}
=\frac{\sqrt{q+\varepsilon}}{\sqrt{q+\varepsilon/a^2}}u_\varepsilon(z).
$$
For $q>0$ and fixed $\varepsilon>0$, this equals $u_\varepsilon(z)$ only when $a=1$. With $\varepsilon=0$, the normalized core is exactly invariant for every $a>0$ wherever $q>0$. The unstabilized scaling argument in <Layer Normalization by Jimmy Lei Ba and others> therefore needs this qualification when used for the stabilized formula.<Layer Normalization by Jimmy Lei Ba and others>[§3, equation 3, and §5.1, equations 6–7: normalization statistics and scaling analysis]{ref}
The same constant controls <layer-normalization sensitivity>. With the <centering matrix> $P=I-d^{-1}\mathbf1\mathbf1^{\mathsf T}$, we have $v=Pz$, $P^{\mathsf T}=P$ and $Pv=v$. Taking a <derivative> gives the <layer-normalization Jacobian>:
$$
Du_\varepsilon(z)=\frac{P}{\sqrt{q+\varepsilon}}-
\frac{vv^{\mathsf T}}{d(q+\varepsilon)^{3/2}}.
$$
The common-offset vector $\mathbf1$ is an <eigenvector> with <eigenvalue> $0$. For $v\ne0$, the radial direction $v$ has <eigenvalue> $\varepsilon/(q+\varepsilon)^{3/2}$; a <vector> orthogonal to both $\mathbf1$ and $v$ has <eigenvalue> $(q+\varepsilon)^{-1/2}$. At a constant input, the <Jacobian matrix> is $P/\sqrt\varepsilon$. These formulas give the <operator norm> bound $\|Du_\varepsilon(z)\|_2\leq\varepsilon^{-1/2}$ for the <Euclidean norm>. Integrating this <derivative> along a <line segment> proves <Lipschitz continuity> of the normalized core with constant $\varepsilon^{-1/2}$. The bound for the full output is multiplied by $\max_i|\gamma_i|$, the <operator norm> of the <diagonal matrix> of its <layer-normalization gain>. For $d\geq2$, the core bound is attained in centered directions at a constant input: smaller $\varepsilon$ permits larger local amplification near zero coordinate <variance>.
For the input family $(-t,0,t)$, the third coordinate of the normalized core is $h_\varepsilon(t)=t/\sqrt{2t^2/3+\varepsilon}$, with <derivative> $h'_\varepsilon(t)=\varepsilon/(2t^2/3+\varepsilon)^{3/2}$. The dashed lines mark $\pm\sqrt{3/2}$; the forward response approaches these values for large positive or negative $t$.
\Image[/layer-normalization-epsilon-response.png]
{title=Original layer-normalization response and sensitivity for $(-t,0,t)$; unit gains and zero biases.}
Back to article page
