= Batch size
{title2=$b$}
\b[Batch size] counts the examples combined in a <mini-batch>. In the simple <stochastic gradient descent> loop where each group produces one parameter update, it controls how many examples contribute to that update. It differs from a <training epoch>, which counts a pass over the available <training data>.<Optimizing Model Parameters in the PyTorch tutorials>[Hyperparameters and Optimization Loop]{ref}
For an original counting example, partition ten examples with nominal batch size $b=4$ and retain the remainder. The actual sizes are $4,4,2$, giving $\lceil10/4\rceil=3$ updates per <training epoch>. Discarding the remainder instead gives two updates and uses only eight examples. More generally the retained-remainder update count is $\lceil n/b\rceil$ for $n$ examples, provided there is one update per group.
Batch size also changes gradient sampling noise. If independently sampled component <gradients> at a fixed parameter have scalar <variance> $\sigma^2$, their <sample mean> has
$$
\operatorname{Var}(G_b)=\frac{\sigma^2}{b}.
$$
For an original exact example, suppose one component gradient is equally likely to be 0 or 2. Its mean is 1 and <variance> is 1. A two-example <mini-batch gradient> takes values $0,1,2$ with probabilities $1/4,1/2,1/4$, giving <variance> $1/2$. This calculation holds parameters fixed and assumes independent sampling; sampling without replacement has a <finite population correction>.
With <gradient accumulation>, several groups may contribute before an update. Two full four-example groups then involve eight examples per update if their gradients are normalized consistently. State whether a reported size counts the processed group or all examples contributing to an update. Lower sampling <variance> alone does not establish faster training for a fixed computation budget.
Back to article page
