 Mini-batch gradient

A mini-batch gradient averages gradients of several sampled components of a finite-sum objective. For $F(x)=n^{-1}{\sum }_{i=1}^n{f}_i(x)$ and batch indices $I_1,\dots ,I_b$,With independent uniform draws at fixed $x$, it is an unbiased stochastic gradient: $\mathbb{E}{G}_b(x)=\nabla F(x)$. Its covariance matrix is $\Sigma (x)/b$, where $\Sigma (x)$ is the covariance of one sampled gradient. The factor follows because centered independent cross terms have expectation zero.

If the indices are sampled uniformly without replacement, the covariance is instead $(n-b)\Sigma (x)/(b(n-1))$ for $n>1$, when $\Sigma$ uses the uniform-population normalization. At $b=n$ this gives the exact full gradient. Larger batches reduce sampling variance but require more component evaluations per update; biased sampling or data dependence requires its own analysis.