= Exploding gradient problem
= Exploding gradient
{synonym}
Repeated differentiation through a deep or recurrent <neural network> can make some <gradient> components extremely large. In the scalar recurrence $h_t=ah_{t-1}$, the derivative with respect to $h_0$ is $a^t$, which grows exponentially if $|a|>1$. Large individual matrix norms alone do not imply such growth, because directions can change or cancel; the full product of step <Jacobian matrices> determines the effect.
Back to article page
