 Exploding gradient problem

Repeated differentiation through a deep or recurrent neural network can make some gradient components extremely large. In the scalar recurrence $h_t=a{h}_{t-1}$, the derivative with respect to $h_0$ is $a^t$, which grows exponentially if $|a|>1$. Large individual matrix norms alone do not imply such growth, because directions can change or cancel; the full product of step Jacobian matrices determines the effect.