= Layer-normalization bias
{title2=$\beta$}
The <layer-normalization bias> is the learned coordinatewise offset $\beta$ applied after the normalized core and <layer-normalization gain>. Writing the core as $u_\varepsilon(z)$,
$$
y_i=\gamma_i u_{\varepsilon,i}(z)+\beta_i.
$$
The offset is a model parameter, rather than an estimate of statistical bias. The PyTorch implementation provides an optional learned additive parameter for each normalized element.<LayerNorm in the PyTorch documentation>[2.15, equation and Variables/bias; Parameters/bias and elementwise_affine]{ref}
An original constant-input example isolates its effect. Choose $z=(3,3,3)$ and any positive <layer-normalization epsilon>. The core is zero, so for $\beta=(1,-1,2)$ the output is exactly $(1,-1,2)$, regardless of the fixed gain. Its coordinate <arithmetic mean> is $2/3$: the output of the full learned layer need not have zero mean even though its normalized core does.
For a scalar loss $\ell$ with incoming output <gradient> $g_i=\partial\ell/\partial y_i$, the <chain rule> gives $\partial\ell/\partial\beta_i=g_i$. For the original loss $\ell=\tfrac12\|y\|_2^2$ in this example, the bias gradient is therefore $(1,-1,2)$. If the same parameter vector is reused at several positions, their gradient contributions add.
An equal shift of every coordinate \i[before] normalization is removed, as shown in <layer normalization is invariant under a common input shift>. The offset here is applied \i[after] normalization and persists. When differentiating only with respect to $z$ while parameters are fixed, $\beta$ contributes zero to the <layer-normalization Jacobian>.
Back to article page
