= Gaussian reparameterization
{c}
{title2=$Z=\mu+\sigma\varepsilon$}
\b[Gaussian reparameterization] represents a <random variable> with a <normal distribution> using a location, a scale and parameter-independent noise:
$$
\varepsilon\sim N(0,1),\qquad Z=\mu+\sigma\varepsilon,\qquad\sigma>0.
$$
Here $\varepsilon$ has the <standard normal distribution>. The resulting $Z$ has <expected value> $\mu$ and <variance> $\sigma^2$. The <probability distribution> is unchanged; the representation separates the random draw from the adjustable parameters. This makes <reparameterization gradients> available for differentiable expectations, including the Gaussian approximate posterior in a <variational autoencoder>.<Stochastic Backpropagation and Approximate Inference in Deep Generative Models by Danilo Jimenez Rezende and others>[§3.2, Gaussian location-scale transformation and equation(10)]{ref}
For a complete example, fix $\sigma$ and consider
$$
F(\mu)=\mathbb E[Z^2]=\mu^2+\sigma^2.
$$
Holding the same noise value fixed while differentiating gives
$$
\frac{\partial}{\partial\mu}(\mu+\sigma\varepsilon)^2=2(\mu+\sigma\varepsilon),\qquad
\mathbb E[2(\mu+\sigma\varepsilon)]=2\mu=F'(\mu).
$$
At $\mu=1$, $\sigma=2$, the particular noise draw $\varepsilon=1/2$ gives $Z=2$ and derivative estimate $4$, while the true derivative is $2$. An individual sample need not equal the derivative; the equality is in <expected value>.
Parameterizing a positive scale as $\sigma=e^\rho$ also gives $\partial Z/\partial\rho=e^\rho\varepsilon$. One can differentiate through the deterministic transformation while leaving the noise law fixed. For a general integrand, exchanging derivative and expectation still requires justified <differentiation under the integral sign>; the representation alone is not a proof of that exchange. Reusing noise when parameters change also connects the construction to <common random numbers>.
Back to article page
