Long short-term memory

Long short-term memory, usually abbreviated LSTM, is a recurrent neural network architecture that carries an explicit memory state through a sequence and learns when to write, retain and expose its contents. It addresses the vanishing gradient problem: in ordinary recurrence, repeated multiplication by small derivatives can make a distant input have almost no effect on the gradient used for learning. Sepp Hochreiter and Jürgen Schmidhuber's 1997 architecture provided a memory-cell path designed to preserve error flow, with gates controlling access.^[ref]

A common simplified modern form, omitting peephole connections, isHere $x_t$ is the input, $c_t$ the memory, $h_t$ the exposed state, $\sigma$ the logistic sigmoid function, $\tanh$ the hyperbolic tangent, and $\odot$ componentwise multiplication. The learned matrices and biases determine an input gate in long short-term memory $i_t$, a forget gate in long short-term memory $f_t$, and an output gate in long short-term memory $o_t$. LSTM: A Search Space Odyssey by Klaus Greff and others gives a more general baseline with peephole connections; the equations above deliberately show the simpler variant.^[ref]

For a scalar memory with writing switched off in the idealized limit and constant retention $f$, $c_n=f^n{c}_0$. Over 100 steps, $f=0.99$ retains about $0.366c_0$, while $f=0.999$ retains about $0.905c_0$. This illustrates how a learned gate can select a retention timescale. Exactly zero or one are limiting idealizations for a finite-input logistic sigmoid function. The product of forget gates describes the direct memory path; it is not the entire recurrent derivative, because gates also depend on earlier states.

The forget gate in long short-term memory was introduced after the original architecture, in Learning to Forget: Continual Prediction with LSTM by Felix A. Gers, Jürgen Schmidhuber and Fred Cummins, to allow old contents to be reset during continuing input streams. Training still uses methods such as backpropagation through time. Artificial delayed-dependency tasks in the original paper demonstrated learning across more than 1,000 time steps in particular experiments; this does not imply unlimited memory or guaranteed success on every long sequence.^[ref][ref]

An LSTM memory cell is the state-carrying unit of a long short-term memory recurrent neural network. It maintains an internal memory $c_t$ whose update is controlled by gates, while an exposed state $h_t$ communicates selected information to the rest of the network. The memory and exposed output are different quantities.^[ref]

In a simplified modern form, omitting peepholes,The LSTM forget gate $f_t$ regulates retention, the LSTM input gate $i_t$ regulates the candidate write $g_t$, and the LSTM output gate $o_t$ regulates exposure. The componentwise product $\odot$ allows different memory coordinates to make different decisions.^[ref]

For a scalar example, $c_{t-1}=0.8$, $f_t=0.9$, $i_t=0.2$ and $g_t=-0.5$ give $c_t=0.62$. If $o_t=0.1$, the output is only $h_t=0.1\tanh (0.62)\approx 0.0551$. A small output can coexist with substantial stored state; it does not demonstrate memory erasure.

The additive state path was designed to help learning across long delays. The original 1997 architecture had input/output gating; adaptive forgetting was introduced later for continuing streams. Preserving a direct gradient path addresses an important learning difficulty but does not guarantee unlimited or perfectly reliable memory for every task.^[ref][ref]

The output gate in long short-term memory controls how much stored memory is exposed as the recurrent output of a long short-term memory network. In a common form,The memory is $c_t$, the exposed state is $h_t$, $\sigma$ is a logistic sigmoid function, and $\tanh$ is the hyperbolic tangent. The output gate in long short-term memory modulates the exposed state rather than directly multiplying the stored memory itself.^[ref]

For example, if $c_t=2$, then $\tanh (c_t)\approx 0.964$. An output gate $o_t=0.05$ yields $h_t\approx 0.0482$, whereas $o_t=0.9$ yields $h_t\approx 0.868$. Both cases can have the same current memory. A small exposed value therefore does not establish that the internal memory has been erased.

Selective exposure lets a network retain information until it becomes useful for a later prediction. The original architecture combined output gating with an input gate in long short-term memory to regulate access to persistent memory. Changing the exposed state may affect later gate decisions, so “does not directly erase” does not mean the output gate has no influence on future memory evolution.^[ref]

The forget gate in long short-term memory controls retention of the previous memory state in a long short-term memory recurrent neural network. In a simplified modern update,Each component of $f_t$ multiplies the corresponding old memory component. Values near one retain it strongly; values near zero suppress it. The input gate in long short-term memory controls the separate new-write term.^[ref]

If a scalar gate stays at $f=0.98$ and the new-write term is zero, then after $n$ steps $c_n=0.98^n{c}_0$. The half-retention time isThis gives an interpretable timescale for a learned multiplicative gate. An exact no-write term is an illustrative condition; finite logistic sigmoid function outputs are strictly between zero and one, though the candidate can itself be zero.

The forget gate was added after the 1997 architecture. Felix A. Gers, Jürgen Schmidhuber and Fred Cummins introduced adaptive forgetting to handle continuing streams where a memory needs to be reset rather than preserved indefinitely. The direct memory-path derivative contains products of forget gates; the full recurrent derivative also includes dependencies through other states and gates.^[ref]

The input gate in long short-term memory controls how much proposed new information is written into the memory state of a long short-term memory recurrent neural network. In a common modern form,Here $g_t$ is the candidate memory update, $i_t$ is the input-gate vector, and $\odot$ means componentwise multiplication. The sigmoid function places each gate component between zero and one. The forget gate in long short-term memory $f_t$ separately controls the existing memory: suppressing a new write does not by itself erase stored information.^[ref]

For a scalar example, let $c_{t-1}=0.7$, $f_t=0.9$, $g_t=-0.5$ and $i_t=0.1$. Then $c_t=0.63-0.05=0.58$. Increasing only $i_t$ to $0.8$ gives $c_t=0.63-0.40=0.23$: a larger gate admits more of the negative candidate. The gate is a multiplier, not an instruction that the memory must increase.

The proposed write and its admission are learned separately. An input may suggest a strong candidate while the gate learns that this timestep should be ignored. This selective writing was part of the original memory-cell strategy for protecting information across long delays, together with gated reading through the output gate in long short-term memory.^[ref]