Transformer model

A sequence model built from stacked transformer blocks, using self-attention to mix information between positions and positionwise feedforward neural networks to transform their representations. Positional encoding supplies sequence-order information. An autoregressive decoder uses causal self-attention; generation remains sequential even when training computations over known sequence positions are parallel.

Information representing sequence position or relative position, supplied to input representations or to attention scores. It can be fixed or learned. Without order information or a position-dependent mask, a self-attention layer using shared projections treats a simultaneous permutation of sequence positions equivariantly.

An attention operation in which queries, keys, and values are projections of one sequence's representations. At each position, learned similarities between its query and permitted keys determine a weighted sum of the corresponding values. Scaled dot-product attention and multi-head attention are standard implementations; causal self-attention restricts which positions can be accessed.

Self-attention whose output at sequence position $t$ may depend only on positions at or before $t$. In a score matrix, set forbidden future-position scores to negative infinity before applying the softmax function; the corresponding weights become zero. This permits parallel computation over a known training sequence without allowing future tokens to influence earlier predictions.

Several independently learned query, key, and value projections apply scaled dot-product attention in parallel. Concatenating their outputs and applying a learned output projection allows different heads to mix sequence information using different learned representations.

With query and key dimension $d_k$, form the score matrix $Q{K}^T/\sqrt{d_k}$, apply the softmax function row by row, and multiply by the values $V$. If query and key coordinates have independent unit variance, their dot product has variance $d_k$; division by $\sqrt{d_k}$ keeps that score scale of order one. Optional additive masks exclude forbidden keys before the softmax function.

A standard transformer block combines multi-head attention with a positionwise feedforward neural network, wrapping each sublayer in a residual connection and layer normalization. An encoder-decoder transformer's decoder also has a cross-attention sublayer. Pre-normalization and post-normalization variants differ in where layer normalization occurs relative to the sublayer and residual connection.