Delta rule ($\Delta W=\eta (t-y)x^{\mathsf{T}}$)

For a linear output $y=Wx$ and squared loss $L=\Vert t-y{\Vert }^2/2$, gradient descent gives $\Delta W=\eta (t-y)x^{\mathsf{T}}$. Indeed $\partial L/\partial W_{ij}=-(t_i-y_i)x_j$. On the same example, $e^{\prime }=(1-\eta \Vert x{\Vert }^2)e$, proving strict error reduction when $e\ne 0$ and $0<\eta \Vert x{\Vert }^2<2$.