= Layer-normalization sensitivity
{title2=$\|Du_\varepsilon\|_2\leq\varepsilon^{-1/2}$}
<Layer-normalization sensitivity> measures how strongly changes in an input can alter its normalized output. For the core $u_\varepsilon$ and the <Euclidean norm>, its local first-order amplification is the <operator norm> $\|Du_\varepsilon(z)\|_2$. The derivation in <layer-normalization epsilon> gives the uniform bound
$$
\|Du_\varepsilon(z)\|_2\leq\varepsilon^{-1/2}.
$$
Integrating the derivative bound yields the corresponding global <Lipschitz continuity> bound. For at least two coordinates, the constant is attained locally in centered directions at constant inputs; it is a worst-case bound, not the amplification at every input.
An original finite perturbation check starts at $z=(7,7)$ and adds $h=(10^{-3},-10^{-3})$. The core at $z$ is zero. With $\varepsilon=10^{-4}$, the norm ratio is
$$
\frac{\|u_\varepsilon(z+h)-u_\varepsilon(z)\|_2}{\|h\|_2}
=\frac{1}{\sqrt{10^{-6}+10^{-4}}}\approx99.5037,
$$
close to the derivative bound $100$. Keeping the same perturbation but taking $\varepsilon=10^{-2}$ gives approximately $9.9995$, with bound $10$. This probes the near-constant-input regime, where the coordinate variance is small compared with epsilon.
For fixed learned gains, the full output has bound $\max_i|\gamma_i|/\sqrt\varepsilon$; a fixed <layer-normalization bias> does not change input differences. The <layer-normalization gain> can therefore amplify or suppress the core response. These are properties of this layer in a specified norm. They do not by themselves bound a complete network, whose other operations and composition also matter. The explicit matrix example in <layer-normalization Jacobian> illustrates why actual amplification can be smaller than the uniform bound.
Back to article page
