Transformer block

A standard transformer block combines multi-head attention with a positionwise feedforward neural network, wrapping each sublayer in a residual connection and layer normalization. An encoder-decoder transformer's decoder also has a cross-attention sublayer. Pre-normalization and post-normalization variants differ in where layer normalization occurs relative to the sublayer and residual connection.