= Long short-term memory
\b[Long short-term memory], usually abbreviated LSTM, is a <recurrent neural network> architecture that carries an explicit memory state through a sequence and learns when to write, retain and expose its contents. It addresses the <vanishing gradient problem>: in ordinary recurrence, repeated multiplication by small derivatives can make a distant input have almost no effect on the gradient used for learning. <Sepp Hochreiter> and <Jürgen Schmidhuber>'s 1997 architecture provided a memory-cell path designed to preserve error flow, with gates controlling access.<Long Short-Term Memory by Sepp Hochreiter and Jürgen Schmidhuber>[Abstract and memory-cell construction: constant error flow and input/output gates.]{ref}
A common simplified modern form, omitting peephole connections, is
$$
\begin{aligned}
i_t&=\sigma(W_i x_t+U_i h_{t-1}+b_i),\\
f_t&=\sigma(W_f x_t+U_f h_{t-1}+b_f),\\
o_t&=\sigma(W_o x_t+U_o h_{t-1}+b_o),\\
g_t&=\tanh(W_g x_t+U_g h_{t-1}+b_g),\\
c_t&=f_t\odot c_{t-1}+i_t\odot g_t,\\
h_t&=o_t\odot\tanh(c_t).
\end{aligned}
$$
Here $x_t$ is the input, $c_t$ the memory, $h_t$ the exposed state, $\sigma$ the logistic <sigmoid function>, $\tanh$ the <hyperbolic tangent>, and $\odot$ componentwise multiplication. The learned matrices and biases determine an <input gate in long short-term memory> $i_t$, a <forget gate in long short-term memory> $f_t$, and an <output gate in long short-term memory> $o_t$. <LSTM: A Search Space Odyssey by Klaus Greff and others> gives a more general baseline with peephole connections; the equations above deliberately show the simpler variant.<LSTM: A Search Space Odyssey by Klaus Greff and others>[Section II and Figure 1: gate roles and baseline update equations.]{ref}
For a scalar memory with writing switched off in the idealized limit and constant retention $f$, $c_n=f^n c_0$. Over 100 steps, $f=0.99$ retains about $0.366c_0$, while $f=0.999$ retains about $0.905c_0$. This illustrates how a learned gate can select a retention timescale. Exactly zero or one are limiting idealizations for a finite-input logistic <sigmoid function>. The product of forget gates describes the direct memory path; it is not the entire recurrent derivative, because gates also depend on earlier states.
The <forget gate in long short-term memory> was introduced after the original architecture, in <Learning to Forget: Continual Prediction with LSTM by Felix A. Gers, Jürgen Schmidhuber and Fred Cummins>, to allow old contents to be reset during continuing input streams. Training still uses methods such as <backpropagation through time>. Artificial delayed-dependency tasks in the original paper demonstrated learning across more than 1,000 time steps in particular experiments; this does not imply unlimited memory or guaranteed success on every long sequence.<Learning to Forget: Continual Prediction with LSTM by Felix A. Gers, Jürgen Schmidhuber and Fred Cummins>[Abstract and adaptive forget-gate motivation.]{ref}<Long Short-Term Memory by Sepp Hochreiter and Jürgen Schmidhuber>[Abstract and long-delay task experiments.]{ref}
Back to article page
