 Input gate in long short-term memory

The input gate in long short-term memory controls how much proposed new information is written into the memory state of a long short-term memory recurrent neural network. In a common modern form,Here $g_t$ is the candidate memory update, $i_t$ is the input-gate vector, and $\odot$ means componentwise multiplication. The sigmoid function places each gate component between zero and one. The forget gate in long short-term memory $f_t$ separately controls the existing memory: suppressing a new write does not by itself erase stored information.^[ref]

For a scalar example, let $c_{t-1}=0.7$, $f_t=0.9$, $g_t=-0.5$ and $i_t=0.1$. Then $c_t=0.63-0.05=0.58$. Increasing only $i_t$ to $0.8$ gives $c_t=0.63-0.40=0.23$: a larger gate admits more of the negative candidate. The gate is a multiplier, not an instruction that the memory must increase.

The proposed write and its admission are learned separately. An input may suggest a strong candidate while the gate learns that this timestep should be ignored. This selective writing was part of the original memory-cell strategy for protecting information across long delays, together with gated reading through the output gate in long short-term memory.^[ref]