Layer normalization ($\mathrm{LN}(z)$)

For $z\in {\mathbb{R}}^d$, define $\bar{z}=d^{-1}{\sum }_j{z}_j$ and $s_z^2=d^{-1}{\sum }_j(z_j-\bar{z}{)}^2$. Layer normalization returns $\gamma \odot (z-\bar{z}1)/\sqrt{s_z^2+\epsilon }+\beta$, with learned coordinatewise scale $\gamma$, shift $\beta$, and fixed $\epsilon >0$. In a transformer block, this normalization is applied across feature coordinates separately within each example and sequence position; it does not average over a batch.