= Input gate in long short-term memory
{title2=$i_t$}
= LSTM input gate
{c}
{synonym}
The \b[input gate in long short-term memory] controls how much proposed new information is written into the memory state of a <long short-term memory> <recurrent neural network>. In a common modern form,
$$
i_t=\sigma(W_i x_t+U_i h_{t-1}+b_i),\qquad
c_t=f_t\odot c_{t-1}+i_t\odot g_t.
$$
Here $g_t$ is the candidate memory update, $i_t$ is the input-gate vector, and $\odot$ means componentwise multiplication. The <sigmoid function> places each gate component between zero and one. The <forget gate in long short-term memory> $f_t$ separately controls the existing memory: suppressing a new write does not by itself erase stored information.<LSTM: A Search Space Odyssey by Klaus Greff and others>[Section II, Figure 1: memory update and input-gate role; equations here omit peephole connections.]{ref}
For a scalar example, let $c_{t-1}=0.7$, $f_t=0.9$, $g_t=-0.5$ and $i_t=0.1$. Then $c_t=0.63-0.05=0.58$. Increasing only $i_t$ to $0.8$ gives $c_t=0.63-0.40=0.23$: a larger gate admits more of the negative candidate. The gate is a multiplier, not an instruction that the memory must increase.
The proposed write and its admission are learned separately. An input may suggest a strong candidate while the gate learns that this timestep should be ignored. This selective writing was part of the original memory-cell strategy for protecting information across long delays, together with gated reading through the <output gate in long short-term memory>.<Long Short-Term Memory by Sepp Hochreiter and Jürgen Schmidhuber>[Memory-cell construction and input/output gates protecting constant error flow.]{ref}
Back to article page
