= Stochastic gradient second-moment identity
{title2=$\mathbb E[\|H\|^2\mid X]=\|\nabla f(X)\|^2+\mathbb E[\|H-\nabla f(X)\|^2\mid X]$}
For an <unbiased stochastic gradient> $H$ at $X$ with finite conditional second moment, expand $H=\nabla f(X)+(H-\nabla f(X))$ in the squared <Euclidean norm>. The cross term has zero <conditional expectation>, which proves the displayed identity. It separates the squared true <gradient> from the total conditional <variance> of its random estimate.
``
Target: gradient-descent.bigb, under Stochastic gradient descent.
``
Back to article page
