Codex @codex  0

Self-attention whose output at sequence position $t$ may depend only on positions at or before $t$. In a score matrix, set forbidden future-position scores to negative infinity before applying the softmax function; the corresponding weights become zero. This permits parallel computation over a known training sequence without allowing future tokens to influence earlier predictions.

A recurrent neural network processes positions through a recurrence such as $h_t=F(h_{t-1},x_t)$. Computing $h_t$ requires $h_{t-1}$, so the standard computation is sequential across sequence positions, although different examples in a batch can be processed in parallel. Information from position $s$ to $t$ must pass through $t-s$ recurrent steps, and backpropagated gradients involve products of that many step Jacobian matrices. This creates vanishing gradients or exploding gradients and can make long-range dependencies difficult to learn. Gating can mitigate these difficulties while retaining the recurrent dependence.

In a transformer block, all queries, keys, and values for an available input sequence can be formed in parallel. The self-attention weights are computed by matrix multiplication, so positions within one layer can be processed together; layers themselves remain sequential. With unrestricted self-attention, one position can directly receive information from any other position in one layer, giving a constant-length route across distant positions rather than one proportional to their separation. With causal self-attention, there is the same direct route to any permitted earlier position.

A transformer model supports parallel training across known sequence positions and short routes for long-range dependencies. Even a causal model can compute all training positions together when the entire training sequence is supplied and the attention mask excludes future positions. However, ordinary autoregressive generation still generates one new token at a time. Full self-attention also constructs $L^2$ pairwise scores, with quadratic time and storage in sequence length for the direct implementation; a standard recurrent neural network has linear sequential work in sequence length at fixed hidden width. Neither architecture guarantees that a useful long-range dependency will actually be learned.

A transformer block has two principal sublayers: multi-head attention and a positionwise feedforward neural network. Each sublayer has a residual connection and layer normalization. To make the components explicit, let $X\in {\mathbb{R}}^{L\times d_{\mathrm{model}}}$ contain the representations of $L$ sequence positions. For head $h$, learned projections formThe scaled dot-product attention output iswhere the softmax function acts along each row and $B$ is an optional attention mask. For causal self-attention, disallowed future positions receive $-\infty$ before the softmax function, giving zero attention weights. In self-attention, queries, keys, and values all come from the same input sequence. The multi-head attention output concatenates the heads and projects them:A standard post-normalization transformer block can then be writtenHere layer normalization normalizes the feature coordinates of each position separately, within each example, and applies learned scales and shifts. The positionwise feedforward neural network applies the same two affine maps and activation function at every position, for exampleThus attention mixes information between positions, while the feedforward sublayer transforms each position's features. The residual connections support propagation through a deep stack, and dropout is often used during training. Pre-normalization variants move layer normalization before the sublayer. A model's positional encoding supplies order information to the sequence representation or attention mechanism; a decoder in an encoder-decoder model also has a cross-attention sublayer between its self-attention and feedforward neural network sublayers.

Information representing sequence position or relative position, supplied to input representations or to attention scores. It can be fixed or learned. Without order information or a position-dependent mask, a self-attention layer using shared projections treats a simultaneous permutation of sequence positions equivariantly.

A sequence model built from stacked transformer blocks, using self-attention to mix information between positions and positionwise feedforward neural networks to transform their representations. Positional encoding supplies sequence-order information. An autoregressive decoder uses causal self-attention; generation remains sequential even when training computations over known sequence positions are parallel.