Past exam of the mathematics course of the University of Cambridge / 2017 / iii / Paper 216 / 5

The most plausible cause is strong dependence between the latent variables and covariance hyperparameters in their joint posterior distribution. In the usual conditionally independent Gaussian process classification model, put $F=(f(x_1),\dots ,f(x_n))$, $w={\sigma }^{-2}$, and $C=w^{-1}{R}_{\tau }$. The probit regression likelihood is $\ell (F)={\prod }_i\Phi ((2y_i-1)F_i)$, while $F\mid w,\tau$ has a multivariate normal distribution with covariance matrix $C$.

For distinct inputs and positive length scales, $R_{\tau }$ is positive definite. Its Gaussian probability density function contributesThe unit-rate exponential prior distribution on $w$ therefore givesIts scale is tightly tied to the current magnitude and shape of $F$. Changing $\tau$ changes the eigenvectors and eigenvalues of its covariance matrix; a latent vector typical under the old covariance may be very atypical under a substantially different one. Holding $F$ fixed can thus restrict the range of a hyperparameter update, and holding the hyperparameters fixed restricts the next latent-function update. Even exact sampling of each full conditional distribution can move slowly along a narrow ridge of the joint posterior distribution.

Binary observations provide limited information about the absolute size of large correctly signed latent values. This can leave substantial scale uncertainty and strengthen the dependence. Long correlation lengths can also make the covariance matrix nearly singular and the latent coordinates highly dependent. These are plausible explanations for a large mixing time, not proofs that every data set causes slow convergence. A blocked Gibbs sampler removes dependence within its chosen blocks, but not dependence between them. Repeated identical inputs should be represented by one shared latent value; otherwise the nominal Gaussian matrix is singular and the displayed inverse formula must be replaced by a representation on its support.

Let $s={\sigma }^2$ and $\vartheta =(s,{\tau }_1,\dots ,{\tau }_m)$. With conditional independence of the binary observations given the latent function, defineThe prior distribution is specified on the precision $s^{-1}$, not on $s$. Its Jacobian determinant givesand the marginal posterior density is proportional to $p(\vartheta )L(\vartheta )$. Working instead with precision or logarithmic coordinates is possible, provided all corresponding Jacobian determinants are included.

Choose an importance sampling density $g_{\vartheta }$ covering the support of $\ell (F){\phi }_{C_{\vartheta }}(F)$, draw $F_1,\dots ,F_M$ independently from it, and use the nonnegative unbiased likelihood estimatorEach summand integrates to $L(\vartheta )$, proving unbiasedness. A simple always-valid choice is the latent Gaussian prior distribution, $g_{\vartheta }={\phi }_{C_{\vartheta }}$, when the covariance matrix is nonsingular. Then the estimator is just the average of the bounded values $\ell (F_j)\in (0,1)$. For duplicate inputs, sample only the distinct latent values and multiply all likelihood contributions at each common input. A Gaussian approximation to the latent posterior distribution, or its mixture with the Gaussian prior distribution, can reduce relative variance while keeping complete support. Its approximation does not replace the exact weights.

The pseudo-marginal Metropolis–Hastings algorithm stores both the current $\vartheta$ and its estimator randomness $u$; write their sampling law as $m_{\vartheta }(u)$. Propose ${\vartheta }^{\prime }\sim q(\vartheta ,\cdot )$, independently generate $u^{\prime }\sim m_{{\vartheta }^{\prime }}$, compute $\hat{L}({\vartheta }^{\prime },u^{\prime })$, and accept the pair withHere $q(\vartheta ,{\vartheta }^{\prime })$ denotes the proposal probability density function from $\vartheta$ to ${\vartheta }^{\prime }$. On rejection retain both the old parameter and the old likelihood estimate. Initialize with a positive estimate. One must not independently refresh the denominator after rejection.

Correctness follows from the extended posterior densityThe pair proposal is $q(\vartheta ,{\vartheta }^{\prime })m_{{\vartheta }^{\prime }}(u^{\prime })$. Its Metropolis–Hastings acceptance probability simplifies to the displayed ratio, since the auxiliary densities cancel. Hence detailed balance holds for $\tilde{\pi }$. Integrating over $u$ gives $p(\vartheta )L(\vartheta )$ by unbiasedness, so the parameter marginal distribution is exactly the intended posterior distribution for any fixed positive $M$. This is exactness of the stationary target, not independent exact samples or a guarantee of rapid convergence. Large relative variance of the likelihood estimates can produce long holding times, so marginalizing $F$ does not by itself guarantee an improvement in mixing time.