Layer-normalization gain ($\gamma$)

The layer-normalization gain is the learned coordinatewise multiplier $\gamma$ applied to the normalized core. With layer-normalization bias $\beta$, the output is $y=\gamma \odot u_{\epsilon }(z)+\beta$, where $\odot$ is the Hadamard product. The multiplier can differ between coordinates and need not be positive.^[ref] It permits the model to rescale the features after normalization.

For an original example, choose $z=(1,1,-2)$ and layer-normalization epsilon $\epsilon =2$. The coordinate mean is zero, the coordinate variance is $2$, and the normalized core is $(1/2,1/2,-1)$. Taking $\gamma =(2,0,-1)$ and zero bias givesThe output coordinate arithmetic mean is $2/3$ and its coordinate variance is $2/9$. Thus the core's centering and variance properties do not automatically survive unequal gains. A negative gain also reverses the sign of its coordinate in this example.

If $g_i=\partial \ell /\partial y_i$ is the incoming loss gradient, then $\partial \ell /\partial {\gamma }_i=g_i{u}_{\epsilon ,i}$. For $\ell =\frac{1}{2}\Vert y{\Vert }_2^2$ above, the gain gradient is $(1/2,0,-1)$. This concerns a derivative with respect to the learned parameter; the layer-normalization Jacobian instead differentiates with respect to the input with parameters fixed.

Setting one ${\gamma }_i$ to zero makes that output coordinate equal to its bias, but does not remove input coordinate $z_i$ from the shared normalization statistics. It can still influence other output coordinates through those statistics. The input derivative is obtained by multiplying the core Jacobian on the left by the diagonal matrix of gains. Layer-normalization epsilon contains the general derivative and bound.