 Normalized coordinate variance in layer normalization

The normalized coordinate variance in layer normalization is the variance across coordinates of the normalized core, with divisor equal to the number of coordinates. This is not a variance across training cases. If the input's centered coordinate variance is $q$ and layer-normalization epsilon is $\epsilon >0$, the core has zero coordinate mean and varianceThe calculation is given in layer-normalization epsilon. The divisor convention agrees with the implementation's correction-zero variance.^[ref]

An original inverse calculation asks how large $q$ must be to attain a desired core variance $0\le r<1$. Solving the displayed relation givesFor chosen $\epsilon =0.04$ and target $r=0.9$, we need $q=0.36$. The two-coordinate input $z=(-0.6,0.6)$ has that variance and normalized core $(-\sqrt{0.9},\sqrt{0.9})$, whose variance is exactly $0.9$. Reaching $r=0.99$ instead requires $q=99\epsilon =3.96$. With fixed positive epsilon, exact unit core variance is never attained at finite $q$.

The restriction applies to the core, before the learned affine transformation. For the first example, uniform layer-normalization gain $\gamma =(2,2)$ and zero layer-normalization bias produce coordinate variance $4(0.9)=3.6$. Unequal gains or biases can change both the mean and variance. A test that assumes the full output must always have unit variance would therefore reject valid layer-normalization computations.