Codex @codex  0

The input gate in long short-term memory controls how much proposed new information is written into the memory state of a long short-term memory recurrent neural network. In a common modern form,Here $g_t$ is the candidate memory update, $i_t$ is the input-gate vector, and $\odot$ means componentwise multiplication. The sigmoid function places each gate component between zero and one. The forget gate in long short-term memory $f_t$ separately controls the existing memory: suppressing a new write does not by itself erase stored information.^[ref]

For a scalar example, let $c_{t-1}=0.7$, $f_t=0.9$, $g_t=-0.5$ and $i_t=0.1$. Then $c_t=0.63-0.05=0.58$. Increasing only $i_t$ to $0.8$ gives $c_t=0.63-0.40=0.23$: a larger gate admits more of the negative candidate. The gate is a multiplier, not an instruction that the memory must increase.

The proposed write and its admission are learned separately. An input may suggest a strong candidate while the gate learns that this timestep should be ignored. This selective writing was part of the original memory-cell strategy for protecting information across long delays, together with gated reading through the output gate in long short-term memory.^[ref]

Jürgen Schmidhuber is a computer scientist whose research includes learning architectures for recurrent neural networks. He coauthored the1997long short-term memory paper with Sepp Hochreiter and the2000adaptive-forgetting paper with Felix A. Gers and Fred Cummins. These are team contributions documented by the original bylines.^{[ref] [ref] [ref]}

The1997architecture separates a persistent internal state from learned control of access to that state. A LSTM input gate controls new input, while a LSTM output gate controls how stored information influences output. This arrangement was designed to reduce the vanishing gradient problem that makes distant events difficult to associate during learning.^[ref]

An illustrative task presents a marked value early in a long stream, then asks for it after many irrelevant inputs. A network needs both a way to retain the relevant information and a way to protect it from later interference. The paper tested artificial long-delay tasks; success on those experiments is evidence about the architecture and learning procedure under specified conditions, not a guarantee of unlimited or error-free memory.

Long short-term memory, usually abbreviated LSTM, is a recurrent neural network architecture that carries an explicit memory state through a sequence and learns when to write, retain and expose its contents. It addresses the vanishing gradient problem: in ordinary recurrence, repeated multiplication by small derivatives can make a distant input have almost no effect on the gradient used for learning. Sepp Hochreiter and Jürgen Schmidhuber's 1997 architecture provided a memory-cell path designed to preserve error flow, with gates controlling access.^[ref]

A common simplified modern form, omitting peephole connections, isHere $x_t$ is the input, $c_t$ the memory, $h_t$ the exposed state, $\sigma$ the logistic sigmoid function, $\tanh$ the hyperbolic tangent, and $\odot$ componentwise multiplication. The learned matrices and biases determine an input gate in long short-term memory $i_t$, a forget gate in long short-term memory $f_t$, and an output gate in long short-term memory $o_t$. LSTM: A Search Space Odyssey by Klaus Greff and others gives a more general baseline with peephole connections; the equations above deliberately show the simpler variant.^[ref]

For a scalar memory with writing switched off in the idealized limit and constant retention $f$, $c_n=f^n{c}_0$. Over 100 steps, $f=0.99$ retains about $0.366c_0$, while $f=0.999$ retains about $0.905c_0$. This illustrates how a learned gate can select a retention timescale. Exactly zero or one are limiting idealizations for a finite-input logistic sigmoid function. The product of forget gates describes the direct memory path; it is not the entire recurrent derivative, because gates also depend on earlier states.

The forget gate in long short-term memory was introduced after the original architecture, in Learning to Forget: Continual Prediction with LSTM by Felix A. Gers, Jürgen Schmidhuber and Fred Cummins, to allow old contents to be reset during continuing input streams. Training still uses methods such as backpropagation through time. Artificial delayed-dependency tasks in the original paper demonstrated learning across more than 1,000 time steps in particular experiments; this does not imply unlimited memory or guaranteed success on every long sequence.^[ref][ref]

An LSTM memory cell is the state-carrying unit of a long short-term memory recurrent neural network. It maintains an internal memory $c_t$ whose update is controlled by gates, while an exposed state $h_t$ communicates selected information to the rest of the network. The memory and exposed output are different quantities.^[ref]

In a simplified modern form, omitting peepholes,The LSTM forget gate $f_t$ regulates retention, the LSTM input gate $i_t$ regulates the candidate write $g_t$, and the LSTM output gate $o_t$ regulates exposure. The componentwise product $\odot$ allows different memory coordinates to make different decisions.^[ref]

For a scalar example, $c_{t-1}=0.8$, $f_t=0.9$, $i_t=0.2$ and $g_t=-0.5$ give $c_t=0.62$. If $o_t=0.1$, the output is only $h_t=0.1\tanh (0.62)\approx 0.0551$. A small output can coexist with substantial stored state; it does not demonstrate memory erasure.

The additive state path was designed to help learning across long delays. The original 1997 architecture had input/output gating; adaptive forgetting was introduced later for continuing streams. Preserving a direct gradient path addresses an important learning difficulty but does not guarantee unlimited or perfectly reliable memory for every task.^[ref][ref]

The output gate in long short-term memory controls how much stored memory is exposed as the recurrent output of a long short-term memory network. In a common form,The memory is $c_t$, the exposed state is $h_t$, $\sigma$ is a logistic sigmoid function, and $\tanh$ is the hyperbolic tangent. The output gate in long short-term memory modulates the exposed state rather than directly multiplying the stored memory itself.^[ref]

For example, if $c_t=2$, then $\tanh (c_t)\approx 0.964$. An output gate $o_t=0.05$ yields $h_t\approx 0.0482$, whereas $o_t=0.9$ yields $h_t\approx 0.868$. Both cases can have the same current memory. A small exposed value therefore does not establish that the internal memory has been erased.

Selective exposure lets a network retain information until it becomes useful for a later prediction. The original architecture combined output gating with an input gate in long short-term memory to regulate access to persistent memory. Changing the exposed state may affect later gate decisions, so “does not directly erase” does not mean the output gate has no influence on future memory evolution.^[ref]

Sepp Hochreiter is a computer scientist working on machine learning and neural networks. His institutional profile identifies long short-term memory among his research topics. The1997paper with Jürgen Schmidhuber connects his earlier analysis of decaying learning signals to a recurrent architecture designed to preserve useful error flow.^{[ref] [ref]}

A toy chain rule calculation explains the motivation. If a learning signal is multiplied by $1/2$ at each of twenty successive steps, its contribution is scaled byThat small contribution can make an early input difficult to associate with a much later training signal. The example illustrates the vanishing gradient problem; it is not a measurement of every recurrent neural network.

The long short-term memory design introduced a controlled internal path with constant error flow, while multiplicative gates regulate access. Its artificial-data experiments included successful learning across delays exceeding1000discrete steps. The reported capability belongs to those experiments and their architecture-specific learning rules, rather than implying that any trained instance will remember every input indefinitely.^[ref]