Layer-normalization sensitivity ($\Vert D{u}_{\epsilon }{\Vert }_2\le {\epsilon }^{-1/2}$)

Layer-normalization sensitivity measures how strongly changes in an input can alter its normalized output. For the core $u_{\epsilon }$ and the Euclidean norm, its local first-order amplification is the operator norm $\Vert D{u}_{\epsilon }(z){\Vert }_2$. The derivation in layer-normalization epsilon gives the uniform boundIntegrating the derivative bound yields the corresponding global Lipschitz continuity bound. For at least two coordinates, the constant is attained locally in centered directions at constant inputs; it is a worst-case bound, not the amplification at every input.

An original finite perturbation check starts at $z=(7,7)$ and adds $h=(10^{-3},-10^{-3})$. The core at $z$ is zero. With $\epsilon =10^{-4}$, the norm ratio isclose to the derivative bound $100$. Keeping the same perturbation but taking $\epsilon =10^{-2}$ gives approximately $9.9995$, with bound $10$. This probes the near-constant-input regime, where the coordinate variance is small compared with epsilon.

For fixed learned gains, the full output has bound ${\max }_i|{\gamma }_i|/\sqrt{\epsilon }$; a fixed layer-normalization bias does not change input differences. The layer-normalization gain can therefore amplify or suppress the core response. These are properties of this layer in a specified norm. They do not by themselves bound a complete network, whose other operations and composition also matter. The explicit matrix example in layer-normalization Jacobian illustrates why actual amplification can be smaller than the uniform bound.