Variance of Gaussian-square finite-difference gradients ($\mathrm{Var}(G)$)

Consider the explicitly specified objectiveand estimate its derivative by a symmetric finite difference with step $h>0$. The choice of shared or independent noise changes the estimator's variance even though both use correct normal distributions at the two parameter values.

With common random numbers, the same $\epsilon$ appears on both sides:Thus $\mathbb{E}[G_{\mathrm{shared}}]=2\mu$ and $\mathrm{Var}(G_{\mathrm{shared}})=4{\sigma }^2$, independent of $h$. Cancellation removes the noisy square term. For this quadratic, the symmetric difference has no truncation bias in exact arithmetic.

For independent standard normals ${\epsilon }_{+},{\epsilon }_{-}$, instead useIf $Y=m+\sigma \epsilon$, expand $Y^2=m^2+2m\sigma \epsilon +{\sigma }^2{\epsilon }^2$. The standard-normal moments $\mathbb{E}{\epsilon }^2=1$, $\mathbb{E}{\epsilon }^3=0$ and $\mathbb{E}{\epsilon }^4=3$ giveAdding the two independent square variances and dividing by $4h^2$ yieldsBoth estimators have the correct expected value $2\mu$, but the independent-noise variance diverges as $h$ tends to zero. At $\mu =\sigma =1$ and $h=0.1$, the variances are $4$ and $302$. An average of $n$ independent replicate estimators divides each variance by $n$.

This is a complete calculation for the stated quadratic, not a universal dominance theorem. At $\mu =0$, $\sigma =1$ and $h=1$, the independent variance is $3$, below the shared value $4$. The covariance principle in common random numbers explains why coupling needs a specified objective and comparison. Very small steps in floating-point evaluation also introduce cancellation concerns that this exact-arithmetic model does not include.