= Amortization gap
The \b[amortization gap] measures how much <evidence lower bound> is lost by using the distribution returned by <amortized inference> instead of the best distribution available for that observation in the same variational family. Hold the generative parameters $\theta$ fixed. Write $\mathcal L_x(q)$ for the <evidence lower bound> and suppose the optimum $q_x^*=\operatorname*{arg\,max}_{q\in\mathcal Q}\mathcal L_x(q)$ is attained. For the shared rule $q_\phi(\cdot\mid x)\in\mathcal Q$,
$$
G_{\rm am}(x)=\mathcal L_x(q_x^*)-\mathcal L_x(q_\phi)\geq0.
$$
The remaining <variational approximation gap> is $G_{\rm approx}(x)=\log p_\theta(x)-\mathcal L_x(q_x^*)$, so the total loss is
$$
\log p_\theta(x)-\mathcal L_x(q_\phi)=G_{\rm approx}(x)+G_{\rm am}(x).
$$
Thus a restrictive <recognition network> or incomplete training can leave an <amortization gap> even when the variational family can represent the exact <posterior distribution>.<Inference Suboptimality in Variational Autoencoders by Chris Cremer and others>[Table1, summary of gap terms; introductory definition of the optimal approximation within its family]{ref}
For an original example, stipulate $p(z\mid x)=N(2x,1)$ for equally weighted observations $x=-1$ and $x=1$. The variational family $\mathcal Q=\{N(m,1):m\in\mathbb R\}$ represents both <posterior distributions> exactly when fitted separately, so $G_{\rm approx}=0$. Restrict the shared rule to $q_b(z\mid x)=N(x+b,1)$: it can adjust the intercept but its slope is fixed at one. The <Kullback-Leibler divergence> between these unit-variance <normal distributions> is $(b-x)^2/2$. Averaging the <amortization gap> over the two observations gives
$$
\overline G_{\rm am}(b)=\frac{(b+1)^2+(b-1)^2}{4}=\frac{b^2+1}{2}.
$$
The best shared intercept $b=0$ still loses $1/2$ nat on average. A current rule with $b=1$ loses one nat: half a nat can be removed by training this rule better, while the remaining half requires changing its fixed-slope restriction. This separates the variational family's expressive power from the shared mapping's capacity and current training state.
In a numerical comparison, a locally improved per-observation distribution need not be the global optimum $q_x^*$. Its improvement supplies a lower bound on the true <amortization gap>; failing to find an improvement does not establish a zero gap. Comparisons must keep both $\theta$ and $\mathcal Q$ fixed.
Back to article page
