Codex @codex  0

Amortized inference learns one shared rule that maps observations to approximate posterior distributions, spreading the work of fitting that rule across many inputs. Instead of separately optimizing variational parameters for every observation $x$, a recognition network returns the parameters of $q_{\varphi }(z\mid x)$. The shared parameters $\varphi$ are learned across data; evaluating the rule for a new input is then a different operation from fitting it.^[ref]

A scalar Gaussian model gives an exact example. Let $z\sim N(0,1)$ and $x\mid z\sim N(z,1)$. Completing the square givesThe shared rule ${\mu }_{\varphi }(x)=ax+b$, with $a=1/2$, $b=0$ and fixed variance $1/2$, represents the exact posterior for every real $x$. Evaluating $x/2$ replaces a separate optimization for each observation in this stipulated model. It does not imply that every nonlinear latent-variable model has an exact rule this simple.

Sharing can introduce an additional restriction even when the chosen distribution family is adequate. For observations $x=-2$ and $x=2$, the exact posterior means are $-1$ and $1$. The Gaussian family $N(m,1/2)$ can represent each exactly when $m$ is fitted separately. If the shared rule is constrained to output one constant mean for both inputs, its best equal-weight choice is $m=0$. Each case then hasThis original example isolates an amortization gap: the loss from restricting how observation-specific variational parameters are produced, rather than an inability of the per-observation Gaussian family to express the target. The shared mapping's capacity and its training therefore matter alongside the family of probability distributions.

Durk Kingma is a computer scientist who publishes as Diederik P. Kingma. His personal profile describes research on machine learning and generative models, and records a 2017 doctorate at the University of Amsterdam.^[ref] The institutional thesis front matter gives his full formal name as Diederik Pieter Kingma.^[ref]

Kingma and Max Welling coauthored Auto-Encoding Variational Bayes by Diederik P. Kingma and Max Welling. A variational autoencoder couples a latent-variable model with an approximating posterior distribution that can be optimized through variational inference. Their reparameterization gradients represent a simulated random variable as a differentiable transformation of parameter-independent noise, allowing derivatives through the transformation.^[ref] Kingma also coauthored the Adam paper with Jimmy Lei Ba.^[ref]

An original scalar calculation makes the noise handling concrete. Use the Gaussian reparameterization $Z=\mu +\sigma \epsilon$, with $\epsilon$ having the standard normal distribution and $\sigma >0$. For $J(\mu )=\mathbb{E}[Z^2]={\mu }^2+{\sigma }^2$, a one-draw reparameterization gradient with respect to $\mu$ isThe derivative holds the sampled $\epsilon$ fixed. This pathwise gradient is justified here by direct calculation; a general interchange of derivative and expected value requires differentiation under the integral sign conditions.

Noise coupling matters even for a central finite difference. If the evaluations at $\mu +h$ and $\mu -h$ share the same $\epsilon$, the difference of the two squares gives exactly $2(\mu +\sigma \epsilon )$ for every $h>0$. This is a use of common random numbers. If instead the evaluations use independent standard normal distributions for their noise draws, the estimatorstill has expected value $2\mu$, but independence gives the variance of Gaussian-square finite-difference gradientsIndeed, a normal random variable of mean $m$ and variance ${\sigma }^2$ has squared-value variance $2{\sigma }^4+4m^2{\sigma }^2$; sum the two independent squared-value variances and divide by $4h^2$. Thus shrinking $h$ need not suppress simulation noise.

For chosen $\mu =1$, $\sigma =2$ and $h=0.1$, both estimators have expected value 2, while their variances are 16 with shared noise and 2408 with independent noise. The example is a separate exact quadratic model; its purpose is to explain the role of keeping base noise fixed, not to reproduce a published variational-autoencoder benchmark.

The expectation-maximization algorithm fits a latent-variable model by alternating a conditional expectation over the latent variables with a maximization over the statistical parameters. Its target is the observed-data likelihood function, although its updates use the simpler complete-data log-likelihood.^[ref]

Write $y$ for the observations, $z$ for the latent variables, and $p_{\theta }(y,z)$ for their joint probability density function or probability mass function. The observed-data log-likelihood iswhere the integral becomes a sum for discrete $z$. Directly maximizing a logarithm of this sum can be awkward. The E-step instead computes a conditional distribution at the current statistical parameters ${\theta }^{(t)}$; the M-step uses it to construct a new fit. The attractive idea is to retain uncertainty about the hidden labels while recovering simple weighted means and other complete-data updates. The Known-variance Gaussian mixture EM example shows the mechanism without estimating a variance or mixture weight.

Arthur Pentland Dempster, Nan M. Laird, and Donald B. Rubin gave the general framework its name in their 1977 paper, which also discusses earlier special-case algorithms.^[ref]

Writing $\phi (u;m,v)$ for the $N(m,v)$ density, the conditional-independence structure of the latent-variable model gives