= Solution
A <transformer block> has two principal sublayers: <multi-head attention> and a positionwise <feedforward neural network>. Each sublayer has a <residual connection> and <layer normalization>. To make the components explicit, let $X\in\mathbb R^{L\times d_{\rm model}}$ contain the representations of $L$ sequence positions. For head $h$, learned projections form
$$
Q_h=XW_h^Q,\qquad K_h=XW_h^K,\qquad V_h=XW_h^V.
$$
The <scaled dot-product attention> output is
$$
A_h=\operatorname{softmax}\left(\frac{Q_hK_h^T}{\sqrt{d_k}}+B\right)V_h,
$$
where the <softmax function> acts along each row and $B$ is an optional attention mask. For <causal self-attention>, disallowed future positions receive $-\infty$ before the <softmax function>, giving zero attention weights. In <self-attention>, queries, keys, and values all come from the same input sequence. The <multi-head attention> output concatenates the heads and projects them:
$$
\operatorname{MHA}(X)=\operatorname{Concat}(A_1,\ldots,A_H)W^O.
$$
A standard post-normalization <transformer block> can then be written
$$
\boxed{U=\operatorname{LN}(X+\operatorname{MHA}(X)),\qquad Y=\operatorname{LN}(U+\operatorname{FFN}(U))}.
$$
Here <layer normalization> normalizes the feature coordinates of each position separately, within each example, and applies learned scales and shifts. The positionwise <feedforward neural network> applies the same two affine maps and <activation function> at every position, for example
$$
\operatorname{FFN}(z)=\rho(zW_1+b_1)W_2+b_2.
$$
Thus \b[attention mixes information between positions, while the feedforward sublayer transforms each position's features]. The <residual connections> support propagation through a deep stack, and <dropout> is often used during training. Pre-normalization variants move <layer normalization> before the sublayer. A model's <positional encoding> supplies order information to the sequence representation or attention mechanism; a decoder in an encoder-decoder model also has a cross-attention sublayer between its <self-attention> and <feedforward neural network> sublayers.
Back to article page
