Self-attention

An attention operation in which queries, keys, and values are projections of one sequence's representations. At each position, learned similarities between its query and permitted keys determine a weighted sum of the corresponding values. Scaled dot-product attention and multi-head attention are standard implementations; causal self-attention restricts which positions can be accessed.

Self-attention whose output at sequence position $t$ may depend only on positions at or before $t$. In a score matrix, set forbidden future-position scores to negative infinity before applying the softmax function; the corresponding weights become zero. This permits parallel computation over a known training sequence without allowing future tokens to influence earlier predictions.

Several independently learned query, key, and value projections apply scaled dot-product attention in parallel. Concatenating their outputs and applying a learned output projection allows different heads to mix sequence information using different learned representations.

With query and key dimension $d_k$, form the score matrix $Q{K}^T/\sqrt{d_k}$, apply the softmax function row by row, and multiply by the values $V$. If query and key coordinates have independent unit variance, their dot product has variance $d_k$; division by $\sqrt{d_k}$ keeps that score scale of order one. Optional additive masks exclude forbidden keys before the softmax function.