Vanishing gradient problem

Repeated differentiation through a deep or recurrent neural network can make gradients associated with distant layers or earlier sequence positions extremely small. A simple mechanism is a product of step Jacobian matrices whose operator norms are all bounded above by some $a<1$: the product operator norm is then at most $a^L$ over $L$ steps by submultiplicativity. This can impede learning long-range dependencies in a recurrent neural network.