= Layer-normalization gain
{title2=$\gamma$}
The <layer-normalization gain> is the learned coordinatewise multiplier $\gamma$ applied to the normalized core. With <layer-normalization bias> $\beta$, the output is $y=\gamma\odot u_\varepsilon(z)+\beta$, where $\odot$ is the <Hadamard product>. The multiplier can differ between coordinates and need not be positive.<LayerNorm in the PyTorch documentation>[2.15, equation; Variables/weight and Parameters/elementwise_affine]{ref} It permits the model to rescale the features after normalization.
For an original example, choose $z=(1,1,-2)$ and <layer-normalization epsilon> $\varepsilon=2$. The coordinate mean is zero, the coordinate variance is $2$, and the normalized core is $(1/2,1/2,-1)$. Taking $\gamma=(2,0,-1)$ and zero bias gives
$$
y=(1,0,1).
$$
The output coordinate <arithmetic mean> is $2/3$ and its coordinate <variance> is $2/9$. Thus the core's centering and variance properties do not automatically survive unequal gains. A negative gain also reverses the sign of its coordinate in this example.
If $g_i=\partial\ell/\partial y_i$ is the incoming loss <gradient>, then $\partial\ell/\partial\gamma_i=g_i u_{\varepsilon,i}$. For $\ell=\tfrac12\|y\|_2^2$ above, the gain gradient is $(1/2,0,-1)$. This concerns a derivative with respect to the learned parameter; the <layer-normalization Jacobian> instead differentiates with respect to the input with parameters fixed.
Setting one $\gamma_i$ to zero makes that output coordinate equal to its bias, but does not remove input coordinate $z_i$ from the shared normalization statistics. It can still influence other output coordinates through those statistics. The input derivative is obtained by multiplying the core Jacobian on the left by the <diagonal matrix> of gains. <Layer-normalization epsilon> contains the general derivative and bound.
Back to article page
