Past exam of the MSc in advanced computer science course of the University of Oxford / 2021 / Bayesian statistical probabilistic programming / 2 / b / Solution

Let $0<p(x)<\infty$, take $q_{\lambda }$ supported where $p(z,x)>0$, and assume the following expectations exist. By the definition of Kullback-Leibler divergence and Bayes' theorem,Therefore the evidence lower bound isNonnegativity of Kullback-Leibler divergence proves $\mathcal{L}(\lambda )\le \log p(x)$. The marginal likelihood $p(x)$ is constant with respect to the variational parameter $\lambda$, so maximizing $\mathcal{L}$ minimizes the stated Kullback-Leibler divergence.

The practical advantage is that evaluating logarithms of joint probabilities and $\log q_{\lambda }$ requires no evaluation of the usually intractable marginal likelihood. One can estimate $\mathcal{L}$ and its gradient with Monte Carlo method. Equality with the log marginal likelihood occurs only when $q_{\lambda }=p(\cdot \mid x)$ almost everywhere. Merely maximizing the evidence lower bound over a restricted family neither computes the exact posterior distribution nor proves that the maximum attains this equality.