 Effective batch size

The effective batch size counts the example contributions combined in one optimizer update, including gradient accumulation and contributions from parallel workers. With $P$ workers, $a$ accumulated groups per worker and $b$ examples in each full group,The PyTorch documentation uses this count to specify when accumulated gradients are unscaled and the optimizer is stepped.^[ref]

For an original example, two workers each process three groups of four examples before updating: $b_{\mathrm{eff}}=2\cdot 3\cdot 4=24$. For an additive per-example loss function, the desired gradient is $24^{-1}{\sum }_{i=1}^{24}\nabla f_i(\theta )$, evaluated at unchanged parameters $\theta$. If each worker accumulates the three group-mean gradients divided by three, then averaging the two workers' results gives that normalization. Summing the workers instead would double the gradient.

Incomplete groups require their actual counts. Group sizes $4,4,2$ on each of two workers give $b_{\mathrm{eff}}=20$, and the combined mini-batch gradient weights each group mean by its example count. Dividing all six group means by six would overweight the two-example groups.

The count describes contributions, not necessarily distinct examples. If both workers process the same twelve example identifiers, there are 24 contributions but only twelve distinct examples. The usual inverse-count reduction of variance requires assumptions about independent random variables; duplication can violate them. Nor does this bookkeeping by itself establish a rule for changing the learning rate.