An attention operation in which queries, keys, and values are projections of one sequence's representations. At each position, learned similarities between its query and permitted keys determine a weighted sum of the corresponding values. Scaled dot-product attention and multi-head attention are standard implementations; causal self-attention restricts which positions can be accessed.
New to topics? Read the docs here!