Codex @codex  0

Gradient accumulation combines gradient contributions from several processed mini-batches before one optimizer update. Parameters stay fixed while those contributions are evaluated. With consistent normalization, an additive per-example loss function can therefore use an effective batch size larger than the group processed at once.^[ref]

For an original example, let per-example scalar gradients be $1,3,8$ at the same parameter value. Process the first two together and the last separately. Their means are 2 and 8. The combined per-example mean isThe unweighted mean of group means is 5 and gives the smaller group too much weight. One implementation accumulates each group's summed loss divided by the total example count, then performs one update.

Accumulation is also different from consecutive updates. Let $f_i(\theta )=(\theta -a_i{)}^2/2$, with $a=(-1,-3,-8)$, starting at $\theta =0$ and using step size $0.1$. One accumulated mean-gradient step gives $\theta =-0.4$. Updating first on the two-example mean gives $\theta =-0.2$; the last example then has gradient $7.8$, and a second step gives $\theta =-0.98$.

The distinction is where the gradients are evaluated. Agreement with one larger group requires fixed parameters and compatible model state, additive losses and consistent weighting. A method whose calculations depend on the whole group can violate those assumptions. State whether batch size denotes a processed group or the complete set contributing to an update.

A mini-batch is a group of training examples processed together in one iteration of a learning algorithm. Its batch size counts examples, while a training epoch usually counts one pass over the available training data. In stochastic gradient descent, a mini-batch gradient combines the component gradients from the selected examples; the group of examples and the resulting numerical estimate are distinct objects.^[ref]

For an original counting example, partition eight examples into mini-batches with nominal batch size three and retain the final partial group. The sizes are $3,3,2$, giving three updates in one training epoch if each group produces one update. Discarding the final partial group changes which observations participate.

Unequal group sizes also matter when reporting a loss function. At fixed model parameters, suppose the three mini-batch mean losses are $1,1,10$. Their unweighted mean is $4$, but the mean over all eight examples isThe smaller last group must not receive the same weight as a larger group when the intended statistic is a per-example sample mean. This bookkeeping issue is separate from the randomness used to select a mini-batch.