= Mini-batch gradient
{title2=$G_b(x)=b^{-1}\sum_j\nabla f_{I_j}(x)$}
A mini-batch gradient averages gradients of several sampled components of a finite-sum objective. For $F(x)=n^{-1}\sum_{i=1}^nf_i(x)$ and batch indices $I_1,\ldots,I_b$,
$$
G_b(x)=\frac1b\sum_{j=1}^b\nabla f_{I_j}(x).
$$
With independent uniform draws at fixed $x$, it is an <unbiased stochastic gradient>: $\mathbb E G_b(x)=\nabla F(x)$. Its <covariance matrix> is $\Sigma(x)/b$, where $\Sigma(x)$ is the covariance of one sampled gradient. The factor follows because centered independent cross terms have expectation zero.
If the indices are sampled uniformly without replacement, the covariance is instead $(n-b)\Sigma(x)/(b(n-1))$ for $n>1$, when $\Sigma$ uses the uniform-population normalization. At $b=n$ this gives the exact full gradient. Larger batches reduce sampling variance but require more component evaluations per update; biased sampling or data dependence requires its own analysis.
Back to article page
