 Long short-term memory

Long short-term memory, usually abbreviated LSTM, is a recurrent neural network architecture that carries an explicit memory state through a sequence and learns when to write, retain and expose its contents. It addresses the vanishing gradient problem: in ordinary recurrence, repeated multiplication by small derivatives can make a distant input have almost no effect on the gradient used for learning. Sepp Hochreiter and Jürgen Schmidhuber's 1997 architecture provided a memory-cell path designed to preserve error flow, with gates controlling access.^[ref]

A common simplified modern form, omitting peephole connections, isHere $x_t$ is the input, $c_t$ the memory, $h_t$ the exposed state, $\sigma$ the logistic sigmoid function, $\tanh$ the hyperbolic tangent, and $\odot$ componentwise multiplication. The learned matrices and biases determine an input gate in long short-term memory $i_t$, a forget gate in long short-term memory $f_t$, and an output gate in long short-term memory $o_t$. LSTM: A Search Space Odyssey by Klaus Greff and others gives a more general baseline with peephole connections; the equations above deliberately show the simpler variant.^[ref]

For a scalar memory with writing switched off in the idealized limit and constant retention $f$, $c_n=f^n{c}_0$. Over 100 steps, $f=0.99$ retains about $0.366c_0$, while $f=0.999$ retains about $0.905c_0$. This illustrates how a learned gate can select a retention timescale. Exactly zero or one are limiting idealizations for a finite-input logistic sigmoid function. The product of forget gates describes the direct memory path; it is not the entire recurrent derivative, because gates also depend on earlier states.

The forget gate in long short-term memory was introduced after the original architecture, in Learning to Forget: Continual Prediction with LSTM by Felix A. Gers, Jürgen Schmidhuber and Fred Cummins, to allow old contents to be reset during continuing input streams. Training still uses methods such as backpropagation through time. Artificial delayed-dependency tasks in the original paper demonstrated learning across more than 1,000 time steps in particular experiments; this does not imply unlimited memory or guaranteed success on every long sequence.^[ref][ref]