Self-attention
= Self-attention
An attention operation in which queries, keys, and values are projections of one sequence's representations. At each position, learned similarities between its query and permitted keys determine a weighted sum of the corresponding values. <Scaled dot-product attention> and <multi-head attention> are standard implementations; <causal self-attention> restricts which positions can be accessed.