= Forget gate in long short-term memory
{title2=$f_t$}
= LSTM forget gate
{c}
{synonym}
The \b[forget gate in long short-term memory] controls retention of the previous memory state in a <long short-term memory> <recurrent neural network>. In a simplified modern update,
$$
f_t=\sigma(W_f x_t+U_f h_{t-1}+b_f),\qquad
c_t=f_t\odot c_{t-1}+i_t\odot g_t.
$$
Each component of $f_t$ multiplies the corresponding old memory component. Values near one retain it strongly; values near zero suppress it. The <input gate in long short-term memory> controls the separate new-write term.<LSTM: A Search Space Odyssey by Klaus Greff and others>[Section II and Figure 1; simplified form here omits peepholes.]{ref}
If a scalar gate stays at $f=0.98$ and the new-write term is zero, then after $n$ steps $c_n=0.98^n c_0$. The half-retention time is
$$
n_{1/2}=\frac{\ln(1/2)}{\ln(0.98)}\approx34.3.
$$
This gives an interpretable timescale for a learned multiplicative gate. An exact no-write term is an illustrative condition; finite logistic <sigmoid function> outputs are strictly between zero and one, though the candidate can itself be zero.
The forget gate was added after the 1997 architecture. <Felix A. Gers>, <Jürgen Schmidhuber> and <Fred Cummins> introduced adaptive forgetting to handle continuing streams where a memory needs to be reset rather than preserved indefinitely. The direct memory-path derivative contains products of forget gates; the full recurrent derivative also includes dependencies through other states and gates.<Learning to Forget: Continual Prediction with LSTM by Felix A. Gers, Jürgen Schmidhuber and Fred Cummins>[Abstract and adaptive forget-gate motivation for continual prediction.]{ref}
Back to article page
