Amortized inference

Amortized inference learns one shared rule that maps observations to approximate posterior distributions, spreading the work of fitting that rule across many inputs. Instead of separately optimizing variational parameters for every observation $x$, a recognition network returns the parameters of $q_{\varphi }(z\mid x)$. The shared parameters $\varphi$ are learned across data; evaluating the rule for a new input is then a different operation from fitting it.^[ref]

A scalar Gaussian model gives an exact example. Let $z\sim N(0,1)$ and $x\mid z\sim N(z,1)$. Completing the square givesThe shared rule ${\mu }_{\varphi }(x)=ax+b$, with $a=1/2$, $b=0$ and fixed variance $1/2$, represents the exact posterior for every real $x$. Evaluating $x/2$ replaces a separate optimization for each observation in this stipulated model. It does not imply that every nonlinear latent-variable model has an exact rule this simple.

Sharing can introduce an additional restriction even when the chosen distribution family is adequate. For observations $x=-2$ and $x=2$, the exact posterior means are $-1$ and $1$. The Gaussian family $N(m,1/2)$ can represent each exactly when $m$ is fitted separately. If the shared rule is constrained to output one constant mean for both inputs, its best equal-weight choice is $m=0$. Each case then hasThis original example isolates an amortization gap: the loss from restricting how observation-specific variational parameters are produced, rather than an inability of the per-observation Gaussian family to express the target. The shared mapping's capacity and its training therefore matter alongside the family of probability distributions.

The amortization gap measures how much evidence lower bound is lost by using the distribution returned by amortized inference instead of the best distribution available for that observation in the same variational family. Hold the generative parameters $\theta$ fixed. Write ${\mathcal{L}}_x(q)$ for the evidence lower bound and suppose the optimum $q_x^{\ast }={\mathrm{argmax}}_{q\in \mathcal{Q}}{\mathcal{L}}_x(q)$ is attained. For the shared rule $q_{\varphi }(\cdot \mid x)\in \mathcal{Q}$,The remaining variational approximation gap is $G_{\mathrm{approx}}(x)=\log p_{\theta }(x)-{\mathcal{L}}_x(q_x^{\ast })$, so the total loss isThus a restrictive recognition network or incomplete training can leave an amortization gap even when the variational family can represent the exact posterior distribution.^[ref]

For an original example, stipulate $p(z\mid x)=N(2x,1)$ for equally weighted observations $x=-1$ and $x=1$. The variational family $\mathcal{Q}=\{N(m,1):m\in \mathbb{R}\}$ represents both posterior distributions exactly when fitted separately, so $G_{\mathrm{approx}}=0$. Restrict the shared rule to $q_b(z\mid x)=N(x+b,1)$: it can adjust the intercept but its slope is fixed at one. The Kullback-Leibler divergence between these unit-variance normal distributions is $(b-x{)}^2/2$. Averaging the amortization gap over the two observations givesThe best shared intercept $b=0$ still loses $1/2$ nat on average. A current rule with $b=1$ loses one nat: half a nat can be removed by training this rule better, while the remaining half requires changing its fixed-slope restriction. This separates the variational family's expressive power from the shared mapping's capacity and current training state.

In a numerical comparison, a locally improved per-observation distribution need not be the global optimum $q_x^{\ast }$. Its improvement supplies a lower bound on the true amortization gap; failing to find an improvement does not establish a zero gap. Comparisons must keep both $\theta$ and $\mathcal{Q}$ fixed.