Past exam of the mathematics course of the University of Cambridge / 2005 / ii / Paper 4 / 6E / a / i

Set $s=|w{|}^2$. The exact Hebbian learning rule givesThus the norm never decreases, and increases whenever the output is nonzero. Because learning is slow relative to input fluctuations, its averaged equation is $\tau \dot{w}=Cw$, where $C=⟨x{x}^T⟩$ is the input second-moment matrix. It is a covariance matrix only if the inputs have zero mean. Diagonalizing the positive semidefinite matrix $C$ gives components $w_j(t)=w_j(0)e^{{\lambda }_{j}t/\tau }$. Hence weights grow without bound whenever the initial vector has a component in an excited positive-eigenvalue direction. A vector entirely in $\ker C$, or identically zero input, is an important stationary exception.