A random vector is an unbiased stochastic gradient at an iterate if
For an adaptive iteration it is often useful to require the stronger historical condition , where contains all information available before drawing the new sample. This states explicitly which information is held fixed in the conditional expectation.
For , a conditionally uniform index gives . If index instead has positive sampling probability , then is the correctly weighted estimate. Omitting these factors generally changes its expectation.
For example, average and . The two sampled gradients are and , whose average is the true gradient . At the minimizer zero, each sample is nonzero even though its mean vanishes; unbiasedness does not mean zero variance.
For an unbiased stochastic gradient at with finite conditional second moment, expand in the squared Euclidean norm. The cross term has zero conditional expectation, which proves the displayed identity. It separates the squared true gradient from the total conditional variance of its random estimate.
Target: gradient-descent.bigb, under Stochastic gradient descent.

Articles by others on the same topic (0)

There are currently no matching articles.