= Variational approximation gap
{title2=$G_{\rm approx}$}
The \b[variational approximation gap] is the part of the <evidence lower bound> deficit that remains even after optimizing separately for an observation within a specified variational family. With the generative model fixed and finite $\log p_\theta(x)$, define
$$
G_{\rm approx}(x)=\log p_\theta(x)-\sup_{q\in\mathcal Q}\mathcal L_x(q)
=\inf_{q\in\mathcal Q}D_{\rm KL}\bigl(q\Vert p_\theta(\cdot\mid x)\bigr).
$$
The equality uses the usual <evidence lower bound> identity for <variational inference>. A supremum allows for a best value that is not attained. The <amortization gap> separately measures failure of the shared inference rule to reach this value.<Inference Suboptimality in Variational Autoencoders by Chris Cremer and others>[Table1; §3.1, equation8, decomposition for the ordinary variational-autoencoder bound]{ref}
An original example shows what independence can cost. Stipulate a centred two-dimensional <multivariate normal distribution> as the <posterior distribution>, with <covariance matrix>
$$
\Sigma=\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix},\qquad |\rho|<1.
$$
Restrict the variational family to centred independent <normal distributions>, $q=N(0,\operatorname{diag}(v_1,v_2))$ with $v_1,v_2>0$. This is a simple <mean-field variational inference> restriction. Writing $s=1-\rho^2$, the <Kullback-Leibler divergence between normal distributions> becomes
$$
D_{\rm KL}(q\Vert p)=\frac12\left[\frac{v_1+v_2}{s}-2+\log\frac{s}{v_1v_2}\right].
$$
Each variance contributes a term $v_i/s-\log v_i$, minimized at $v_i=s$. Consequently
$$
G_{\rm approx}=-\frac12\log(1-\rho^2).
$$
For $\rho=0.6$, the best independent variances are both $0.64$, and the unavoidable deficit is about $0.2231$ nat. Matching both marginal variances to one instead would give about $0.3394$ nat. The reverse <Kullback-Leibler divergence> therefore chooses narrower independent marginals here, while still failing to represent the correlation.
Enlarging the family to include the full <covariance matrix> permits $q=p$ and reduces this example's gap to zero. More generally, inclusion $\mathcal Q_1\subseteq\mathcal Q_2$ cannot increase the optimum approximation gap at a fixed model. Merely training the same shared rule longer cannot remove a strictly positive family restriction. Numerical improvements are not proofs that the infimum has been reached.
Back to article page
