= Effective batch size
{title2=$b_{\rm eff}$}
The <effective batch size> counts the example contributions combined in one <optimizer update>, including <gradient accumulation> and contributions from parallel workers. With $P$ workers, $a$ accumulated groups per worker and $b$ examples in each full group,
$$
b_{\rm eff}=Pab.
$$
The <PyTorch> documentation uses this count to specify when accumulated gradients are unscaled and the optimizer is stepped.<Automatic Mixed Precision examples in the PyTorch documentation>[PyTorch 2.15, Gradient accumulation subsection, effective-batch formula and example loop]{ref}
For an original example, two workers each process three groups of four examples before updating: $b_{\rm eff}=2\cdot3\cdot4=24$. For an additive per-example <loss function>, the desired <gradient> is $24^{-1}\sum_{i=1}^{24}\nabla f_i(\theta)$, evaluated at unchanged parameters $\theta$. If each worker accumulates the three group-mean gradients divided by three, then averaging the two workers' results gives that normalization. Summing the workers instead would double the <gradient>.
Incomplete groups require their actual counts. Group sizes $4,4,2$ on each of two workers give $b_{\rm eff}=20$, and the combined <mini-batch gradient> weights each group mean by its example count. Dividing all six group means by six would overweight the two-example groups.
The count describes contributions, not necessarily distinct examples. If both workers process the same twelve example identifiers, there are 24 contributions but only twelve distinct examples. The usual inverse-count reduction of <variance> requires assumptions about <independent random variables>; duplication can violate them. Nor does this bookkeeping by itself establish a rule for changing the <learning rate>.
Back to article page
