Hopfield network

A recurrent network with symmetric interactions and an energy function. In the binary version, units take values $\pm 1$, self-couplings are zero, and asynchronous threshold updates can retrieve stored patterns from incomplete or corrupted cues. Fixed points are local energy minima, which need not be the intended memories or global minima.

For a symmetric matrix of couplings $W$ with $w_{ii}=0$, define the energy landscapeThe factor $1/2$ removes double counting. The local input is $h_k={\sum }_{j\ne k}{w}_{kj}{x}_j-{\theta }_k$. Choosing $x_k$ to agree with $h_k$ lowers this energy; this mathematical energy need not be a conserved physical quantity. Its pairwise form connects the model to the Ising model; competing couplings also motivate the comparison with a spin glass. Here retrieval seeks a useful nearby local minimum, so reaching the global minimum is not the goal.

John Hopfield's 1982 computer study stored random binary strings in neural networks of 30 and 100 units. Couplings were prepared from the stored patterns; asynchronous updates supplied the computational apparatus. Starting at each nominal memory, the readout counted Hamming distances after settling as storage load varied. Corrupted starting patterns separately tested recovery. Varying the load and cue damage exposed interference between memories. These simulations supported collective retrieval, not a measurement of biological memory capacity.^[ref]

For symmetric zero-diagonal weights and zero thresholds, update one binary unit to the sign of its current field, retaining it at a zero field. The displayed energy change is negative on every actual flip. There are finitely many states, so a fair asynchronous update schedule eventually reaches a fixed point that is a minimum against single-unit flips. Synchronous updates can instead cycle, and local minima need not be global minima.

To obtain the displayed identity, subtract the two energies. Terms not involving $x_k$ cancel, and symmetry combines the two copies of each pair. With thresholds included the result is $\Delta E=-(x_k^{\prime }-x_k)h_k$. The zero-threshold expression above is its specialization. At $h_k=0$, keeping the old state rules out motion along equal-energy states. Finiteness then excludes infinitely many strict decreases, while fairness ensures that an unstable unit cannot be ignored forever. A local minimum here means no single flip lowers energy; equality directions can remain.

The asynchronous condition matters even for two units: with $w_{12}=w_{21}=1$ and zero thresholds, simultaneous updates alternate $(1,-1)$ and $(-1,1)$ forever. Either single-unit update instead reaches an aligned stable state.

For $M$ binary units and desired patterns ${\xi }^{\mu }$, use these symmetric weights and set $w_{ii}=0$. For one pattern, the field at that pattern is $(M-1){\xi }_i/M$, so it is stable. Several patterns add cross-talk terms; storage by this rule alone does not guarantee stability of every arbitrary collection of patterns.

Writing $m_{\mu }=M^{-1}{\sum }_j{\xi }_j^{\mu }{x}_j$ givesfor $P$ stored patterns. The subtraction removes the forbidden self-couplings. It makes explicit how the desired association competes with all other stored patterns. An unintended stable state can therefore arise even though every update obeys asynchronous Hopfield energy descent.

Store only $\xi =(1,-1,1,-1,1)$ with $M=5$, zero thresholds, and Hebbian memory weights for a Hopfield network. If $d$ bits differ from $\xi$, their Hamming distance is $d$ and ${\sum }_i{\xi }_i{x}_i=M-2d$. Squaring this sum and subtracting the $M$ diagonal terms yields the displayed energy. Thus $d=0,1,2,3,4,5$ gives $E=-2,-2/5,2/5,2/5,-2/5,-2$.

For the cue $x=(-1,1,1,-1,1)$, the first unit has input $h_1=2/5$ and flips to $1$. The second then has $h_2=-4/5$ and flips to $-1$, completing recall. The energy drops $2/5\to -2/5\to -2$. Every cue with at most two wrong bits is corrected under a fair asynchronous schedule: a wrong bit sees the correct majority among the other four, while a correct bit either sees agreement or retains its value at a tie. The opposite pattern $-\xi$ is also stored implicitly, so cues with at least three errors retrieve it. This is a limitation of this symmetric one-pattern construction.