= Layer-normalization Jacobian
{title2=$Du_\varepsilon(z)$}
The <layer-normalization Jacobian> describes the first-order response of the normalized core to a change in its input. Let $u_\varepsilon$ be the core defined in <layer-normalization epsilon>, with $\varepsilon>0$. Its <Jacobian matrix> $J(z)=Du_\varepsilon(z)$ satisfies
$$
u_\varepsilon(z+h)=u_\varepsilon(z)+J(z)h+o(\|h\|_2).
$$
The closed formula and its derivation are given in <layer-normalization epsilon>. A matrix entry $J_{ij}$ measures the local effect of changing input coordinate $j$ on normalized output coordinate $i$.
An original two-coordinate calculation takes $z=(-1,1)^{\mathsf T}$ and $\varepsilon=3$. Here the centered variance is $1$ and the normalized core is $(-1/2,1/2)^{\mathsf T}$. Substitution into the established derivative formula gives
$$
J(z)=\frac{3}{16}\begin{pmatrix}1&-1\\-1&1\end{pmatrix}.
$$
Consequently
$$
J(z)\begin{pmatrix}1\\1\end{pmatrix}=0,\qquad
J(z)\begin{pmatrix}1\\-1\end{pmatrix}=\begin{pmatrix}3/8\\-3/8\end{pmatrix}.
$$
The common-offset direction has no effect; the centered contrast direction changes the output. This is a local linear response, rather than an exact rule for arbitrary finite changes.
For the complete output with fixed <layer-normalization gain> and <layer-normalization bias>, the input Jacobian is $\operatorname{diag}(\gamma)J(z)$. In the example, $\gamma=(0,2)$ gives
$$
D_z\operatorname{LN}_\varepsilon(z)=\begin{pmatrix}0&0\\-3/8&3/8\end{pmatrix}.
$$
The first output is suppressed, yet the first input still affects the second output. This distinguishes an output gain from removing an input feature. Parameter derivatives with respect to $\gamma$ and $\beta$ are separate derivatives. <Layer-normalization sensitivity> measures the input Jacobian's amplification through an <operator norm>.
Back to article page
