= Solution
A <recurrent neural network> processes positions through a recurrence such as $h_t=F(h_{t-1},x_t)$. Computing $h_t$ requires $h_{t-1}$, so the standard computation is sequential across sequence positions, although different examples in a batch can be processed in parallel. Information from position $s$ to $t$ must pass through $t-s$ recurrent steps, and backpropagated <gradients> involve products of that many step <Jacobian matrices>. This creates <vanishing gradients> or <exploding gradients> and can make long-range dependencies difficult to learn. Gating can mitigate these difficulties while retaining the recurrent dependence.
In a <transformer block>, all queries, keys, and values for an available input sequence can be formed in parallel. The <self-attention> weights are computed by <matrix multiplication>, so positions within one layer can be processed together; layers themselves remain sequential. With unrestricted <self-attention>, one position can directly receive information from any other position in one layer, giving a constant-length route across distant positions rather than one proportional to their separation. With <causal self-attention>, there is the same direct route to any permitted earlier position.
\b[A <transformer model> supports parallel training across known sequence positions and short routes for long-range dependencies.] Even a causal model can compute all training positions together when the entire training sequence is supplied and the attention mask excludes future positions. However, ordinary autoregressive generation still generates one new token at a time. Full <self-attention> also constructs $L^2$ pairwise scores, with quadratic time and storage in sequence length for the direct implementation; a standard <recurrent neural network> has linear sequential work in sequence length at fixed hidden width. Neither architecture guarantees that a useful long-range dependency will actually be learned.
Back to article page
