Scaled dot-product attention
= Scaled dot-product attention
{title2=$\operatorname{softmax}(QK^T/\sqrt{d_k})V$}
With query and key dimension $d_k$, form the score <matrix> $QK^T/\sqrt{d_k}$, apply the <softmax function> row by row, and multiply by the values $V$. If query and key coordinates have independent unit <variance>, their dot product has <variance> $d_k$; division by $\sqrt{d_k}$ keeps that score scale of order one. Optional additive masks exclude forbidden keys before the <softmax function>.