 Forget gate in long short-term memory

The forget gate in long short-term memory controls retention of the previous memory state in a long short-term memory recurrent neural network. In a simplified modern update,Each component of $f_t$ multiplies the corresponding old memory component. Values near one retain it strongly; values near zero suppress it. The input gate in long short-term memory controls the separate new-write term.^[ref]

If a scalar gate stays at $f=0.98$ and the new-write term is zero, then after $n$ steps $c_n=0.98^n{c}_0$. The half-retention time isThis gives an interpretable timescale for a learned multiplicative gate. An exact no-write term is an illustrative condition; finite logistic sigmoid function outputs are strictly between zero and one, though the candidate can itself be zero.

The forget gate was added after the 1997 architecture. Felix A. Gers, Jürgen Schmidhuber and Fred Cummins introduced adaptive forgetting to handle continuing streams where a memory needs to be reset rather than preserved indefinitely. The direct memory-path derivative contains products of forget gates; the full recurrent derivative also includes dependencies through other states and gates.^[ref]