 Layer-normalization bias

The layer-normalization bias is the learned coordinatewise offset $\beta$ applied after the normalized core and layer-normalization gain. Writing the core as $u_{\epsilon }(z)$,The offset is a model parameter, rather than an estimate of statistical bias. The PyTorch implementation provides an optional learned additive parameter for each normalized element.^[ref]

An original constant-input example isolates its effect. Choose $z=(3,3,3)$ and any positive layer-normalization epsilon. The core is zero, so for $\beta =(1,-1,2)$ the output is exactly $(1,-1,2)$, regardless of the fixed gain. Its coordinate arithmetic mean is $2/3$: the output of the full learned layer need not have zero mean even though its normalized core does.

For a scalar loss $\ell$ with incoming output gradient $g_i=\partial \ell /\partial y_i$, the chain rule gives $\partial \ell /\partial {\beta }_i=g_i$. For the original loss $\ell =\frac{1}{2}\Vert y{\Vert }_2^2$ in this example, the bias gradient is therefore $(1,-1,2)$. If the same parameter vector is reused at several positions, their gradient contributions add.

An equal shift of every coordinate before normalization is removed, as shown in layer normalization is invariant under a common input shift. The offset here is applied after normalization and persists. When differentiating only with respect to $z$ while parameters are fixed, $\beta$ contributes zero to the layer-normalization Jacobian.