= Variance of Gaussian-square finite-difference gradients
{title2=$\operatorname{Var}(G)$}
Consider the explicitly specified objective
$$
F(\mu)=\mathbb E[(\mu+\sigma\varepsilon)^2]=\mu^2+\sigma^2,\qquad \varepsilon\sim N(0,1),\quad\sigma>0,
$$
and estimate its derivative by a symmetric <finite difference> with step $h>0$. The choice of shared or independent noise changes the estimator's <variance> even though both use correct <normal distributions> at the two parameter values.
With <common random numbers>, the same $\varepsilon$ appears on both sides:
$$
G_{\mathrm{shared}}=\frac{(\mu+h+\sigma\varepsilon)^2-(\mu-h+\sigma\varepsilon)^2}{2h}
=2\mu+2\sigma\varepsilon.
$$
Thus $\mathbb E[G_{\mathrm{shared}}]=2\mu$ and \b[$\operatorname{Var}(G_{\mathrm{shared}})=4\sigma^2$], independent of $h$. Cancellation removes the noisy square term. For this quadratic, the symmetric difference has no truncation bias in exact arithmetic.
For independent standard normals $\varepsilon_+,\varepsilon_-$, instead use
$$
G_{\mathrm{ind}}=\frac{(\mu+h+\sigma\varepsilon_+)^2-(\mu-h+\sigma\varepsilon_-)^2}{2h}.
$$
If $Y=m+\sigma\varepsilon$, expand $Y^2=m^2+2m\sigma\varepsilon+\sigma^2\varepsilon^2$. The standard-normal moments $\mathbb E\varepsilon^2=1$, $\mathbb E\varepsilon^3=0$ and $\mathbb E\varepsilon^4=3$ give
$$
\operatorname{Var}(Y^2)=4m^2\sigma^2+2\sigma^4.
$$
Adding the two independent square variances and dividing by $4h^2$ yields
$$
\operatorname{Var}(G_{\mathrm{ind}})=\frac{\sigma^4+2\mu^2\sigma^2}{h^2}+2\sigma^2.
$$
Both estimators have the correct <expected value> $2\mu$, but the independent-noise variance diverges as $h$ tends to zero. At $\mu=\sigma=1$ and $h=0.1$, the variances are $4$ and $302$. An average of $n$ independent replicate estimators divides each variance by $n$.
This is a complete calculation for the stated quadratic, not a universal dominance theorem. At $\mu=0$, $\sigma=1$ and $h=1$, the independent variance is $3$, below the shared value $4$. The covariance principle in <common random numbers> explains why coupling needs a specified objective and comparison. Very small steps in floating-point evaluation also introduce cancellation concerns that this exact-arithmetic model does not include.
Back to article page
