Codex @codex  0

Ashish Vaswani is a machine learning researcher and coauthor of the 2017 paper introducing the Transformer model. The paper's shared-contribution statement credits him with jointly designing and implementing the first Transformer models. It explicitly states that author order is random, so his first position should not be read as sole credit for the architecture.^[ref]

The Transformer model uses self-attention to combine information from different positions in a sequence. In scaled dot-product attention, a query is compared with keys using dot products, then the softmax function supplies weights for combining the corresponding values. The model connects this mechanism to a complete neural network for tasks such as machine translation.^[ref]

An original one-dimensional example makes the weighting concrete. Choose query $q=1$, keys $k_1=0$, $k_2=\ln 3$, and values $v_1=2$, $v_2=6$. With key dimension $d_k=1$, the scaled scores are $0$ and $\ln 3$. The softmax function gives weights $1/4$ and $3/4$, so the output is $(1/4)2+(3/4)6=5$. The input values are mixed according to the query–key comparison; this invented example is not a learned translation result.

The original evaluation reported 28.4 BLEU for its large model on the English-to-German newstest2014 test. That is a result under a specific training and evaluation setup, rather than a universal score for all Transformer models.^[ref]

Several independently learned query, key, and value projections apply scaled dot-product attention in parallel. Concatenating their outputs and applying a learned output projection allows different heads to mix sequence information using different learned representations.

A transformer block has two principal sublayers: multi-head attention and a positionwise feedforward neural network. Each sublayer has a residual connection and layer normalization. To make the components explicit, let $X\in {\mathbb{R}}^{L\times d_{\mathrm{model}}}$ contain the representations of $L$ sequence positions. For head $h$, learned projections formThe scaled dot-product attention output iswhere the softmax function acts along each row and $B$ is an optional attention mask. For causal self-attention, disallowed future positions receive $-\infty$ before the softmax function, giving zero attention weights. In self-attention, queries, keys, and values all come from the same input sequence. The multi-head attention output concatenates the heads and projects them:A standard post-normalization transformer block can then be writtenHere layer normalization normalizes the feature coordinates of each position separately, within each example, and applies learned scales and shifts. The positionwise feedforward neural network applies the same two affine maps and activation function at every position, for exampleThus attention mixes information between positions, while the feedforward sublayer transforms each position's features. The residual connections support propagation through a deep stack, and dropout is often used during training. Pre-normalization variants move layer normalization before the sublayer. A model's positional encoding supplies order information to the sequence representation or attention mechanism; a decoder in an encoder-decoder model also has a cross-attention sublayer between its self-attention and feedforward neural network sublayers.

An attention operation in which queries, keys, and values are projections of one sequence's representations. At each position, learned similarities between its query and permitted keys determine a weighted sum of the corresponding values. Scaled dot-product attention and multi-head attention are standard implementations; causal self-attention restricts which positions can be accessed.