Layer normalization
= Layer normalization
{title2=$\mathrm{LN}(z)$}
For $z\in\mathbb R^d$, define $\bar z=d^{-1}\sum_jz_j$ and $s_z^2=d^{-1}\sum_j(z_j-\bar z)^2$. Layer normalization returns $\gamma\odot(z-\bar z\mathbf1)/\sqrt{s_z^2+\varepsilon}+\beta$, with learned coordinatewise scale $\gamma$, shift $\beta$, and fixed $\varepsilon>0$. In a <transformer block>, this normalization is applied across feature coordinates separately within each example and sequence position; it does not average over a batch.