= Vanishing gradient problem
= Vanishing gradient
{synonym}
Repeated differentiation through a deep or recurrent <neural network> can make <gradients> associated with distant layers or earlier sequence positions extremely small. A simple mechanism is a product of step <Jacobian matrices> whose <operator norms> are all bounded above by some $a<1$: the product <operator norm> is then at most $a^L$ over $L$ steps by submultiplicativity. This can impede learning long-range dependencies in a <recurrent neural network>.
Back to article page
