Gaussian reparameterization ($Z=\mu +\sigma \epsilon$)

Gaussian reparameterization represents a random variable with a normal distribution using a location, a scale and parameter-independent noise:Here $\epsilon$ has the standard normal distribution. The resulting $Z$ has expected value $\mu$ and variance ${\sigma }^2$. The probability distribution is unchanged; the representation separates the random draw from the adjustable parameters. This makes reparameterization gradients available for differentiable expectations, including the Gaussian approximate posterior in a variational autoencoder.^[ref]

For a complete example, fix $\sigma$ and considerHolding the same noise value fixed while differentiating givesAt $\mu =1$, $\sigma =2$, the particular noise draw $\epsilon =1/2$ gives $Z=2$ and derivative estimate $4$, while the true derivative is $2$. An individual sample need not equal the derivative; the equality is in expected value.

Parameterizing a positive scale as $\sigma =e^{\rho }$ also gives $\partial Z/\partial \rho =e^{\rho }\epsilon$. One can differentiate through the deterministic transformation while leaving the noise law fixed. For a general integrand, exchanging derivative and expectation still requires justified differentiation under the integral sign; the representation alone is not a proof of that exchange. Reusing noise when parameters change also connects the construction to common random numbers.

Consider the explicitly specified objectiveand estimate its derivative by a symmetric finite difference with step $h>0$. The choice of shared or independent noise changes the estimator's variance even though both use correct normal distributions at the two parameter values.

With common random numbers, the same $\epsilon$ appears on both sides:Thus $\mathbb{E}[G_{\mathrm{shared}}]=2\mu$ and $\mathrm{Var}(G_{\mathrm{shared}})=4{\sigma }^2$, independent of $h$. Cancellation removes the noisy square term. For this quadratic, the symmetric difference has no truncation bias in exact arithmetic.

For independent standard normals ${\epsilon }_{+},{\epsilon }_{-}$, instead useIf $Y=m+\sigma \epsilon$, expand $Y^2=m^2+2m\sigma \epsilon +{\sigma }^2{\epsilon }^2$. The standard-normal moments $\mathbb{E}{\epsilon }^2=1$, $\mathbb{E}{\epsilon }^3=0$ and $\mathbb{E}{\epsilon }^4=3$ giveAdding the two independent square variances and dividing by $4h^2$ yieldsBoth estimators have the correct expected value $2\mu$, but the independent-noise variance diverges as $h$ tends to zero. At $\mu =\sigma =1$ and $h=0.1$, the variances are $4$ and $302$. An average of $n$ independent replicate estimators divides each variance by $n$.

This is a complete calculation for the stated quadratic, not a universal dominance theorem. At $\mu =0$, $\sigma =1$ and $h=1$, the independent variance is $3$, below the shared value $4$. The covariance principle in common random numbers explains why coupling needs a specified objective and comparison. Very small steps in floating-point evaluation also introduce cancellation concerns that this exact-arithmetic model does not include.