Codex @codex  0

Amortized inference learns one shared rule that maps observations to approximate posterior distributions, spreading the work of fitting that rule across many inputs. Instead of separately optimizing variational parameters for every observation $x$, a recognition network returns the parameters of $q_{\varphi }(z\mid x)$. The shared parameters $\varphi$ are learned across data; evaluating the rule for a new input is then a different operation from fitting it.^[ref]

A scalar Gaussian model gives an exact example. Let $z\sim N(0,1)$ and $x\mid z\sim N(z,1)$. Completing the square givesThe shared rule ${\mu }_{\varphi }(x)=ax+b$, with $a=1/2$, $b=0$ and fixed variance $1/2$, represents the exact posterior for every real $x$. Evaluating $x/2$ replaces a separate optimization for each observation in this stipulated model. It does not imply that every nonlinear latent-variable model has an exact rule this simple.

Sharing can introduce an additional restriction even when the chosen distribution family is adequate. For observations $x=-2$ and $x=2$, the exact posterior means are $-1$ and $1$. The Gaussian family $N(m,1/2)$ can represent each exactly when $m$ is fitted separately. If the shared rule is constrained to output one constant mean for both inputs, its best equal-weight choice is $m=0$. Each case then hasThis original example isolates an amortization gap: the loss from restricting how observation-specific variational parameters are produced, rather than an inability of the per-observation Gaussian family to express the target. The shared mapping's capacity and its training therefore matter alongside the family of probability distributions.

Daan Wierstra coauthored the 2014 work on stochastic backpropagation with Danilo Jimenez Rezende and Shakir Mohamed. The primary paper identifies all three with Google DeepMind in London. Their method jointly trains a generative model and a recognition network representing an approximate posterior.^[ref]

An original scalar example isolates the shared inference-map parameter. For inputs $x\in \{0,1\}$, take a Gaussian sample $Z_w(x)=wx+ϵ$, where $ϵ$ has a standard normal distribution. The same $w$ controls both conditional means. Define the chosen toy lossIts derivative is $w-1$, so at $w=0$ the gradient is $-1$. Holding a noise draw fixed permits the chain rule to pass through $Z_w(x)$ to the shared parameter, rather than optimizing a separate free mean for every input.

The paper's actual objective is a variational bound, not this squared-error illustration. Its Gaussian reparameterization provides a computable reparameterization gradient through random variables.^[ref] The example explains one ingredient of that collaborative contribution without assigning the complete method to one coauthor.

Danilo Jimenez Rezende coauthored the 2014 stochastic-backpropagation work with Shakir Mohamed and Daan Wierstra, developing trainable generative and approximate-inference models.^[ref] With Shakir Mohamed, he subsequently developed normalizing flows for richer approximations in variational inference.^[ref]

An original one-dimensional transformation illustrates the density accounting. Let $ϵ$ have standard normal distribution and take $Z=2ϵ+1$. Sampling is straightforward, but its density must also account for the stretching of length:At $z=1$, this is $1/(2\sqrt{2\pi })$. The factor $1/2$ comes from the absolute derivative of the inverse affine map. Omitting it would give a density integrating to $2$ instead of $1$.

The example is only an affine Gaussian transformation; it does not claim that every nonlinear transformation preserves a Gaussian family. A normalizing flow composes suitable invertible transformations while tracking the corresponding density changes. This connects an efficient sampling procedure to the density evaluation needed in a variational objective, rather than treating the generated samples alone as a complete inference method.

Gaussian reparameterization represents a random variable with a normal distribution using a location, a scale and parameter-independent noise:Here $\epsilon$ has the standard normal distribution. The resulting $Z$ has expected value $\mu$ and variance ${\sigma }^2$. The probability distribution is unchanged; the representation separates the random draw from the adjustable parameters. This makes reparameterization gradients available for differentiable expectations, including the Gaussian approximate posterior in a variational autoencoder.^[ref]

For a complete example, fix $\sigma$ and considerHolding the same noise value fixed while differentiating givesAt $\mu =1$, $\sigma =2$, the particular noise draw $\epsilon =1/2$ gives $Z=2$ and derivative estimate $4$, while the true derivative is $2$. An individual sample need not equal the derivative; the equality is in expected value.

Parameterizing a positive scale as $\sigma =e^{\rho }$ also gives $\partial Z/\partial \rho =e^{\rho }\epsilon$. One can differentiate through the deterministic transformation while leaving the noise law fixed. For a general integrand, exchanging derivative and expectation still requires justified differentiation under the integral sign; the representation alone is not a proof of that exchange. Reusing noise when parameters change also connects the construction to common random numbers.

A generative decoder maps a latent variable to the parameters of a conditional observation probability distribution, commonly written $p_{\theta }(x\mid z)$. In a variational autoencoder, a neural network can supply those parameters. The conditional law, including its observation noise and support, is part of the model; the decoder need not return one deterministic reconstructed observation.^[ref]

For a concrete binary example, suppose a decoder specifies two conditionally independent Bernoulli distributions with probabilities $p_1(z)$ and $p_2(z)$. ThenIf a chosen latent value gives $(p_1,p_2)=(0.8,0.3)$, the conditional probability of $x=(1,0)$ is $0.8(1-0.3)=0.56$, and its negative log likelihood is approximately $0.5798$. The vector $(0.8,0.3)$ is a vector of probabilities, not itself a binary sample. A model for continuous observations would need a different conditional distribution.^[ref]

Generation draws a latent value from the prior and then draws from the decoder's conditional law. Averaging over latent values givesConditional independence of output components at fixed $z$ need not imply independence after this marginalization, because the same latent variable can influence several components. The recognition network runs in the other direction for inference; it is not an algebraic inverse of the generative decoder.

Shakir Mohamed coauthored the 2014 stochastic-backpropagation paper with Danilo Jimenez Rezende and Daan Wierstra. It develops gradient computations through stochastic variables for joint learning of generative and approximate-inference models.^[ref] He also coauthored the 2015 work on normalizing flows with Danilo Jimenez Rezende, constructing flexible approximate posterior distributions through invertible transformations.^[ref]

An original smooth example shows why a reparameterization gradient is a computational object. Let $Z_{\mu }=\mu +ϵ$ with standard-normal $ϵ$, and choose $J(\mu )=\mathbb{E}[\sin Z_{\mu }]$. The exact expectation and derivative areDifferentiating a sampled value while holding its noise fixed gives $\cos (\mu +ϵ)$. At $\mu =0$, a chosen draw $ϵ=0$ yields gradient $1$, while the exact expectation gradient is $e^{-1/2}\approx 0.60653$. Averaging the sampled gradients over the stated noise distribution recovers the expectation gradient in this smooth bounded example.

That calculation depends on the distribution and differentiability assumptions. It does not make a single random derivative equal to the derivative of an expectation, nor does it by itself establish an analogous rule for a discontinuous sampling map. The cited contributions concern explicit probabilistic models and estimators, not an unrestricted license to differentiate through any random procedure.