 Layer-normalization epsilon

The positive constant $\epsilon$ in layer normalization regularizes division by the coordinate standard deviation. For $d$ coordinates, write $\mu =d^{-1}{\sum }_i{z}_i$, $v=z-\mu 1$ and $q=d^{-1}\Vert v{\Vert }_2^2$. The normalized core and the full output areHere $\odot$ is the Hadamard product, and $\gamma ,\beta$ are fixed when studying the dependence on $z$. This placement of $\epsilon$, the divisor $d$ in the coordinate variance, and the elementwise layer-normalization gain and layer-normalization bias define the stabilized convention used here.^[ref] If all coordinates of $z$ are equal, then $v=0$ and $q=0$: positive $\epsilon$ makes the normalized core well defined and equal to 0, so the full output is $\beta$.

Positive $\epsilon$ also changes the normalization away from constant inputs. The arithmetic mean of the normalized core is 0, but its normalized coordinate variance in layer normalization isThus it approaches 1 when $q/\epsilon$ is large, equals $1/2$ when $q=\epsilon$, and approaches 0 when $q/\epsilon$ approaches 0. This formula describes the normalized core; learned, unequal coordinate gains or biases can change the coordinate arithmetic mean and variance of the full output. For $z=(1,2,3{)}^{\mathsf{T}}$, changing $\epsilon$ from $1/3$ to $4/3$ changes the core from $(-1,0,1{)}^{\mathsf{T}}$ to $(-1,0,1{)}^{\mathsf{T}}/\sqrt{2}$.

Although layer normalization is invariant under a common input shift, a fixed positive $\epsilon$ prevents exact positive scaling invariance on a nonconstant input. For $a>0$,For $q>0$ and fixed $\epsilon >0$, this equals $u_{\epsilon }(z)$ only when $a=1$. With $\epsilon =0$, the normalized core is exactly invariant for every $a>0$ wherever $q>0$. The unstabilized scaling argument in Layer Normalization by Jimmy Lei Ba and others therefore needs this qualification when used for the stabilized formula.^[ref]

The same constant controls layer-normalization sensitivity. With the centering matrix $P=I-d^{-1}11^{\mathsf{T}}$, we have $v=Pz$, $P^{\mathsf{T}}=P$ and $Pv=v$. Taking a derivative gives the layer-normalization Jacobian:The common-offset vector $1$ is an eigenvector with eigenvalue $0$. For $v\ne 0$, the radial direction $v$ has eigenvalue $\epsilon /(q+\epsilon {)}^{3/2}$; a vector orthogonal to both $1$ and $v$ has eigenvalue $(q+\epsilon {)}^{-1/2}$. At a constant input, the Jacobian matrix is $P/\sqrt{\epsilon }$. These formulas give the operator norm bound $\Vert D{u}_{\epsilon }(z){\Vert }_2\le {\epsilon }^{-1/2}$ for the Euclidean norm. Integrating this derivative along a line segment proves Lipschitz continuity of the normalized core with constant ${\epsilon }^{-1/2}$. The bound for the full output is multiplied by ${\max }_i|{\gamma }_i|$, the operator norm of the diagonal matrix of its layer-normalization gain. For $d\ge 2$, the core bound is attained in centered directions at a constant input: smaller $\epsilon$ permits larger local amplification near zero coordinate variance.

For the input family $(-t,0,t)$, the third coordinate of the normalized core is $h_{\epsilon }(t)=t/\sqrt{2t^2/3+\epsilon }$, with derivative $h_{\epsilon }^{\prime }(t)=\epsilon /(2t^2/3+\epsilon {)}^{3/2}$. The dashed lines mark $\pm \sqrt{3/2}$; the forward response approaches these values for large positive or negative $t$.