 Layer-normalization Jacobian

The layer-normalization Jacobian describes the first-order response of the normalized core to a change in its input. Let $u_{\epsilon }$ be the core defined in layer-normalization epsilon, with $\epsilon >0$. Its Jacobian matrix $J(z)=D{u}_{\epsilon }(z)$ satisfiesThe closed formula and its derivation are given in layer-normalization epsilon. A matrix entry $J_{ij}$ measures the local effect of changing input coordinate $j$ on normalized output coordinate $i$.

An original two-coordinate calculation takes $z=(-1,1{)}^{\mathsf{T}}$ and $\epsilon =3$. Here the centered variance is $1$ and the normalized core is $(-1/2,1/2{)}^{\mathsf{T}}$. Substitution into the established derivative formula givesConsequentlyThe common-offset direction has no effect; the centered contrast direction changes the output. This is a local linear response, rather than an exact rule for arbitrary finite changes.

For the complete output with fixed layer-normalization gain and layer-normalization bias, the input Jacobian is $\mathrm{diag}(\gamma )J(z)$. In the example, $\gamma =(0,2)$ givesThe first output is suppressed, yet the first input still affects the second output. This distinguishes an output gain from removing an input feature. Parameter derivatives with respect to $\gamma$ and $\beta$ are separate derivatives. Layer-normalization sensitivity measures the input Jacobian's amplification through an operator norm.