= Mini-batch
A \b[mini-batch] is a group of training examples processed together in one iteration of a learning algorithm. Its <batch size> counts examples, while a <training epoch> usually counts one pass over the available <training data>. In <stochastic gradient descent>, a <mini-batch gradient> combines the component <gradients> from the selected examples; the group of examples and the resulting numerical estimate are distinct objects.<Optimizing Model Parameters in the PyTorch tutorials>[Hyperparameters and Optimization Loop: batch size, epochs and updates.]{ref}
For an original counting example, partition eight examples into <mini-batches> with nominal <batch size> three and retain the final partial group. The sizes are $3,3,2$, giving three updates in one <training epoch> if each group produces one update. Discarding the final partial group changes which observations participate.
Unequal group sizes also matter when reporting a <loss function>. At fixed model parameters, suppose the three <mini-batch> mean losses are $1,1,10$. Their unweighted mean is $4$, but the mean over all eight examples is
$$
\frac{3(1)+3(1)+2(10)}8=\frac{26}8=3.25.
$$
The smaller last group must not receive the same weight as a larger group when the intended statistic is a per-example <sample mean>. This bookkeeping issue is separate from the randomness used to select a <mini-batch>.
Back to article page
