Mini-batch

A mini-batch is a group of training examples processed together in one iteration of a learning algorithm. Its batch size counts examples, while a training epoch usually counts one pass over the available training data. In stochastic gradient descent, a mini-batch gradient combines the component gradients from the selected examples; the group of examples and the resulting numerical estimate are distinct objects.^[ref]

For an original counting example, partition eight examples into mini-batches with nominal batch size three and retain the final partial group. The sizes are $3,3,2$, giving three updates in one training epoch if each group produces one update. Discarding the final partial group changes which observations participate.

Unequal group sizes also matter when reporting a loss function. At fixed model parameters, suppose the three mini-batch mean losses are $1,1,10$. Their unweighted mean is $4$, but the mean over all eight examples isThe smaller last group must not receive the same weight as a larger group when the intended statistic is a per-example sample mean. This bookkeeping issue is separate from the randomness used to select a mini-batch.

Batch size counts the examples combined in a mini-batch. In the simple stochastic gradient descent loop where each group produces one parameter update, it controls how many examples contribute to that update. It differs from a training epoch, which counts a pass over the available training data.^[ref]

For an original counting example, partition ten examples with nominal batch size $b=4$ and retain the remainder. The actual sizes are $4,4,2$, giving $\lceil 10/4\rceil =3$ updates per training epoch. Discarding the remainder instead gives two updates and uses only eight examples. More generally the retained-remainder update count is $\lceil n/b\rceil$ for $n$ examples, provided there is one update per group.

Batch size also changes gradient sampling noise. If independently sampled component gradients at a fixed parameter have scalar variance ${\sigma }^2$, their sample mean hasFor an original exact example, suppose one component gradient is equally likely to be 0 or 2. Its mean is 1 and variance is 1. A two-example mini-batch gradient takes values $0,1,2$ with probabilities $1/4,1/2,1/4$, giving variance $1/2$. This calculation holds parameters fixed and assumes independent sampling; sampling without replacement has a finite population correction.

With gradient accumulation, several groups may contribute before an update. Two full four-example groups then involve eight examples per update if their gradients are normalized consistently. State whether a reported size counts the processed group or all examples contributing to an update. Lower sampling variance alone does not establish faster training for a fixed computation budget.

Gradient accumulation combines gradient contributions from several processed mini-batches before one optimizer update. Parameters stay fixed while those contributions are evaluated. With consistent normalization, an additive per-example loss function can therefore use an effective batch size larger than the group processed at once.^[ref]

For an original example, let per-example scalar gradients be $1,3,8$ at the same parameter value. Process the first two together and the last separately. Their means are 2 and 8. The combined per-example mean isThe unweighted mean of group means is 5 and gives the smaller group too much weight. One implementation accumulates each group's summed loss divided by the total example count, then performs one update.

Accumulation is also different from consecutive updates. Let $f_i(\theta )=(\theta -a_i{)}^2/2$, with $a=(-1,-3,-8)$, starting at $\theta =0$ and using step size $0.1$. One accumulated mean-gradient step gives $\theta =-0.4$. Updating first on the two-example mean gives $\theta =-0.2$; the last example then has gradient $7.8$, and a second step gives $\theta =-0.98$.

The distinction is where the gradients are evaluated. Agreement with one larger group requires fixed parameters and compatible model state, additive losses and consistent weighting. A method whose calculations depend on the whole group can violate those assumptions. State whether batch size denotes a processed group or the complete set contributing to an update.

The effective batch size counts the example contributions combined in one optimizer update, including gradient accumulation and contributions from parallel workers. With $P$ workers, $a$ accumulated groups per worker and $b$ examples in each full group,The PyTorch documentation uses this count to specify when accumulated gradients are unscaled and the optimizer is stepped.^[ref]

For an original example, two workers each process three groups of four examples before updating: $b_{\mathrm{eff}}=2\cdot 3\cdot 4=24$. For an additive per-example loss function, the desired gradient is $24^{-1}{\sum }_{i=1}^{24}\nabla f_i(\theta )$, evaluated at unchanged parameters $\theta$. If each worker accumulates the three group-mean gradients divided by three, then averaging the two workers' results gives that normalization. Summing the workers instead would double the gradient.

Incomplete groups require their actual counts. Group sizes $4,4,2$ on each of two workers give $b_{\mathrm{eff}}=20$, and the combined mini-batch gradient weights each group mean by its example count. Dividing all six group means by six would overweight the two-example groups.

The count describes contributions, not necessarily distinct examples. If both workers process the same twelve example identifiers, there are 24 contributions but only twelve distinct examples. The usual inverse-count reduction of variance requires assumptions about independent random variables; duplication can violate them. Nor does this bookkeeping by itself establish a rule for changing the learning rate.