= Normalized coordinate variance in layer normalization
{title2=$q/(q+\varepsilon)$}
The <normalized coordinate variance in layer normalization> is the variance across coordinates of the normalized core, with divisor equal to the number of coordinates. This is not a variance across training cases. If the input's centered coordinate variance is $q$ and <layer-normalization epsilon> is $\varepsilon>0$, the core has zero coordinate mean and variance
$$
r=\frac{q}{q+\varepsilon}.
$$
The calculation is given in <layer-normalization epsilon>. The divisor convention agrees with the implementation's correction-zero variance.<LayerNorm in the PyTorch documentation>[2.15, equation and correction=0 convention]{ref}
An original inverse calculation asks how large $q$ must be to attain a desired core variance $0\leq r<1$. Solving the displayed relation gives
$$
q=\frac{r}{1-r}\varepsilon.
$$
For chosen $\varepsilon=0.04$ and target $r=0.9$, we need $q=0.36$. The two-coordinate input $z=(-0.6,0.6)$ has that variance and normalized core $(-\sqrt{0.9},\sqrt{0.9})$, whose variance is exactly $0.9$. Reaching $r=0.99$ instead requires $q=99\varepsilon=3.96$. With fixed positive epsilon, exact unit core variance is never attained at finite $q$.
The restriction applies to the core, before the learned affine transformation. For the first example, uniform <layer-normalization gain> $\gamma=(2,2)$ and zero <layer-normalization bias> produce coordinate variance $4(0.9)=3.6$. Unequal gains or biases can change both the mean and variance. A test that assumes the full output must always have unit variance would therefore reject valid layer-normalization computations.
Back to article page
