= Gradient accumulation
\b[Gradient accumulation] combines gradient contributions from several processed <mini-batches> before one optimizer update. Parameters stay fixed while those contributions are evaluated. With consistent normalization, an additive per-example <loss function> can therefore use an <effective batch size> larger than the group processed at once.<Automatic Mixed Precision examples in the PyTorch documentation>[Gradient accumulation subsection and example loop]{ref}
For an original example, let per-example scalar <gradients> be $1,3,8$ at the same parameter value. Process the first two together and the last separately. Their means are 2 and 8. The combined per-example mean is
$$
\boxed{G=\frac{2(2)+1(8)}3=4.}
$$
The unweighted mean of group means is 5 and gives the smaller group too much weight. One implementation accumulates each group's summed loss divided by the total example count, then performs one update.
Accumulation is also different from consecutive updates. Let $f_i(\theta)=(\theta-a_i)^2/2$, with $a=(-1,-3,-8)$, starting at $\theta=0$ and using <step size> $0.1$. One accumulated mean-gradient step gives $\theta=-0.4$. Updating first on the two-example mean gives $\theta=-0.2$; the last example then has gradient $7.8$, and a second step gives $\theta=-0.98$.
The distinction is where the gradients are evaluated. Agreement with one larger group requires fixed parameters and compatible model state, additive losses and consistent weighting. A method whose calculations depend on the whole group can violate those assumptions. State whether <batch size> denotes a processed group or the complete set contributing to an update.
Back to article page
